2023-10-03T19:59:23.013 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T19:59:23.014 DEBUG:teuthology.run:Teuthology command: teuthology --name rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --owner scheduled_rraja@teuthology --verbose --archive /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828 -- /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828/orig.config.yaml 2023-10-03T19:59:23.047 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T19:59:23.116 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828 branch: wip-ajarr-63009-only-reacquire-kickstart description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7409828' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-ajarr-63009-only-reacquire-kickstart ceph: conf: client: rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ajarr-check-addr-non-zero sha1: 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6885 sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-ajarr-check-addr-non-zero suite_path: /home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa suite_relpath: qa suite_repo: https://github.com/ajarr/ceph suite_sha1: 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 targets: smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5dENvZ8CjFh1oP9HEOOvUxtCMq4Isbx8hO++REGaHaB5KJwIPB64PWkzDAYeBogfIJrNMGYZDotx2PmdKyH+w= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-03_19:52:41 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-03T19:59:23.117 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa; will attempt to use it 2023-10-03T19:59:23.118 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks 2023-10-03T19:59:23.118 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-03T19:59:23.119 INFO:teuthology.task.internal:Checking packages... 2023-10-03T19:59:23.141 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad' 2023-10-03T19:59:23.141 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-03T19:59:23.141 INFO:teuthology.packaging:ref: None 2023-10-03T19:59:23.141 INFO:teuthology.packaging:tag: None 2023-10-03T19:59:23.141 INFO:teuthology.packaging:branch: wip-ajarr-63009-only-reacquire-kickstart 2023-10-03T19:59:23.141 INFO:teuthology.packaging:sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T19:59:23.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-ajarr-63009-only-reacquire-kickstart 2023-10-03T19:59:23.360 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T19:59:23.360 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-03T19:59:23.367 INFO:teuthology.task.internal:no buildpackages task found 2023-10-03T19:59:23.367 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-03T19:59:23.390 INFO:teuthology.task.internal:Saving configuration 2023-10-03T19:59:23.403 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-03T19:59:23.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T19:59:23.435 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 19:53:35.524529', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBbnMLN58QldCDNzyNcYRydI+5oMtHbCw0i1eW7UiLYn'} 2023-10-03T19:59:23.435 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-03T19:59:23.441 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-03T19:59:23.441 INFO:teuthology.run_tasks:Running task console_log... 2023-10-03T19:59:23.481 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3421302af0>, signals=[15]) 2023-10-03T19:59:23.482 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-03T19:59:23.488 INFO:teuthology.task.internal:Opening connections... 2023-10-03T19:59:23.488 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T19:59:23.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T19:59:23.565 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-03T19:59:23.571 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2023-10-03T19:59:23.674 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2023-10-03T19:59:23.675 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2023-10-03T19:59:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-03T19:59:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="Ubuntu" 2023-10-03T19:59:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="22.04" 2023-10-03T19:59:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-03T19:59:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_CODENAME=jammy 2023-10-03T19:59:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:ID=ubuntu 2023-10-03T19:59:23.721 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE=debian 2023-10-03T19:59:23.721 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-03T19:59:23.721 INFO:teuthology.orchestra.run.smithi167.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-03T19:59:23.721 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-03T19:59:23.721 INFO:teuthology.orchestra.run.smithi167.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-03T19:59:23.721 INFO:teuthology.orchestra.run.smithi167.stdout:UBUNTU_CODENAME=jammy 2023-10-03T19:59:23.722 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2023-10-03T19:59:23.745 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-03T19:59:23.753 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-03T19:59:23.761 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-03T19:59:23.761 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2023-10-03T19:59:23.771 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-03T19:59:23.777 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-03T19:59:23.777 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2023-10-03T19:59:23.817 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-03T19:59:23.849 INFO:teuthology.run_tasks:Running task kernel... 2023-10-03T19:59:23.863 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-03T19:59:23.863 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-03T19:59:23.863 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-03T19:59:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T19:59:23.868 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-10-03T19:59:23.916 INFO:teuthology.orchestra.run.smithi167.stdout:5.15.0-56-generic 2023-10-03T19:59:23.916 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.15.0-56-generic 2023-10-03T19:59:23.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2023-10-03T19:59:24.038 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-03T19:59:24.311 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T19:59:24.312 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-03T19:59:24.375 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-03T19:59:24.727 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-03T19:59:24.732 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [822 kB] 2023-10-03T19:59:24.866 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [497 kB] 2023-10-03T19:59:25.047 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,022 kB] 2023-10-03T19:59:25.098 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [340 kB] 2023-10-03T19:59:25.121 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [171 kB] 2023-10-03T19:59:25.132 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB] 2023-10-03T19:59:25.133 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-03T19:59:25.135 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [899 kB] 2023-10-03T19:59:25.169 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [229 kB] 2023-10-03T19:59:25.202 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB] 2023-10-03T19:59:25.203 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-03T19:59:25.204 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [915 kB] 2023-10-03T19:59:25.216 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [145 kB] 2023-10-03T19:59:25.226 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-03T19:59:25.226 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [560 kB] 2023-10-03T19:59:25.238 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [147 kB] 2023-10-03T19:59:25.244 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-03T19:59:25.258 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [786 kB] 2023-10-03T19:59:25.408 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [657 kB] 2023-10-03T19:59:25.449 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [144 kB] 2023-10-03T19:59:25.832 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-10-03T19:59:25.839 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-03T19:59:25.839 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-03T19:59:25.854 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-03T19:59:25.858 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-03T19:59:25.978 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [987 kB] 2023-10-03T19:59:26.088 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [216 kB] 2023-10-03T19:59:26.107 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.9 kB] 2023-10-03T19:59:26.109 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-03T19:59:26.109 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-03T19:59:26.112 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-03T19:59:26.113 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-03T19:59:26.113 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-03T19:59:26.116 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-03T19:59:26.119 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-03T19:59:26.284 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-03T19:59:26.366 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-03T19:59:26.372 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-03T19:59:26.491 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-03T19:59:26.542 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-03T19:59:30.800 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 9,246 kB in 2s (3,799 kB/s) 2023-10-03T19:59:32.073 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T19:59:32.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-03T19:59:32.148 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T19:59:32.332 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-03T19:59:32.332 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-03T19:59:32.481 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T19:59:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T19:59:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T19:59:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T19:59:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-03T19:59:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: python2.7-minimal 2023-10-03T19:59:32.482 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T19:59:32.483 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2023-10-03T19:59:32.483 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-10-03T19:59:32.483 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-generic linux-image-5.15.0-84-generic 2023-10-03T19:59:32.483 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-5.15.0-84-generic linux-modules-extra-5.15.0-84-generic 2023-10-03T19:59:32.484 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2023-10-03T19:59:32.484 INFO:teuthology.orchestra.run.smithi167.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-03T19:59:32.484 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2023-10-03T19:59:32.485 INFO:teuthology.orchestra.run.smithi167.stdout: thermald 2023-10-03T19:59:32.533 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2023-10-03T19:59:32.533 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-10-03T19:59:32.534 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.15.0-84-generic linux-modules-5.15.0-84-generic 2023-10-03T19:59:32.534 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-extra-5.15.0-84-generic 2023-10-03T19:59:32.535 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages will be upgraded: 2023-10-03T19:59:32.535 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-03T19:59:32.604 INFO:teuthology.orchestra.run.smithi167.stdout:3 upgraded, 5 newly installed, 0 to remove and 266 not upgraded. 2023-10-03T19:59:32.604 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 113 MB of archives. 2023-10-03T19:59:32.604 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-03T19:59:32.604 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-84-generic amd64 5.15.0-84.93 [22.5 MB] 2023-10-03T19:59:33.962 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-84-generic amd64 5.15.0-84.93 [11.5 MB] 2023-10-03T19:59:34.291 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-84-generic amd64 5.15.0-84.93 [63.9 MB] 2023-10-03T19:59:35.637 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.84.81 [1,698 B] 2023-10-03T19:59:35.637 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.84.81 [2,476 B] 2023-10-03T19:59:35.713 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84 all 5.15.0-84.93 [12.3 MB] 2023-10-03T19:59:35.908 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84-generic amd64 5.15.0-84.93 [2,853 kB] 2023-10-03T19:59:35.955 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.84.81 [2,330 B] 2023-10-03T19:59:36.350 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 113 MB in 3s (33.1 MB/s) 2023-10-03T19:59:36.709 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-5.15.0-84-generic. 2023-10-03T19:59:38.706 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-03T19:59:38.711 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../0-linux-modules-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T19:59:38.781 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T19:59:40.858 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-image-5.15.0-84-generic. 2023-10-03T19:59:40.880 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../1-linux-image-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T19:59:40.960 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T19:59:41.412 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-84-generic. 2023-10-03T19:59:41.432 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T19:59:41.470 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T19:59:47.638 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../3-linux-generic_5.15.0.84.81_amd64.deb ... 2023-10-03T19:59:47.765 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-10-03T19:59:48.106 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.84.81_amd64.deb ... 2023-10-03T19:59:48.236 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-10-03T19:59:48.497 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.15.0-84. 2023-10-03T19:59:48.519 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../5-linux-headers-5.15.0-84_5.15.0-84.93_all.deb ... 2023-10-03T19:59:48.563 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-10-03T19:59:54.248 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.15.0-84-generic. 2023-10-03T19:59:54.272 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../6-linux-headers-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T19:59:54.315 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T19:59:56.867 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.84.81_amd64.deb ... 2023-10-03T19:59:56.995 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-10-03T19:59:57.345 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-10-03T19:59:57.467 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T19:59:57.593 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-generic (5.15.0.84.81) ... 2023-10-03T19:59:57.721 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T19:59:59.067 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-03T19:59:59.067 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-03T19:59:59.067 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-84-generic 2023-10-03T19:59:59.067 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-84-generic 2023-10-03T19:59:59.241 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T20:00:00.692 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T20:00:02.094 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-generic (5.15.0.84.81) ... 2023-10-03T20:00:02.211 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-generic (5.15.0.84.81) ... 2023-10-03T20:00:02.338 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T20:00:02.407 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-03T20:00:02.408 INFO:teuthology.orchestra.run.smithi167.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic 2023-10-03T20:00:16.476 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-03T20:00:16.476 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub' 2023-10-03T20:00:16.494 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-03T20:00:16.514 INFO:teuthology.orchestra.run.smithi167.stdout:Generating grub configuration file ... 2023-10-03T20:00:16.854 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-10-03T20:00:16.870 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-10-03T20:00:17.242 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-03T20:00:17.249 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-03T20:00:17.325 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-03T20:00:17.330 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-03T20:00:17.468 INFO:teuthology.orchestra.run.smithi167.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-03T20:00:17.468 INFO:teuthology.orchestra.run.smithi167.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-03T20:00:17.469 INFO:teuthology.orchestra.run.smithi167.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-03T20:00:17.512 INFO:teuthology.orchestra.run.smithi167.stdout:done 2023-10-03T20:00:17.946 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 20 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2023-10-03T20:00:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.15.0.84.81 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2023-10-03T20:00:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2023-10-03T20:00:17.971 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-84-generic 2023-10-03T20:00:17.971 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-03T20:00:17.972 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-03T20:00:17.972 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-03T20:00:17.972 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2023-10-03T20:00:17.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2023-10-03T20:00:18.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-03T20:00:18.260 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-03T20:00:22.083 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T20:00:22.098 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-03T20:00:22.115 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-03T20:00:23.282 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:00:23.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-03T20:00:23.361 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:00:23.544 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-03T20:00:23.545 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-03T20:00:23.703 INFO:teuthology.orchestra.run.smithi167.stdout:linux-image-generic is already the newest version (5.15.0.84.81). 2023-10-03T20:00:23.703 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T20:00:23.703 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T20:00:23.703 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T20:00:23.703 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T20:00:23.704 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-03T20:00:23.704 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-03T20:00:23.704 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T20:00:23.744 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 0 newly installed, 0 to remove and 266 not upgraded. 2023-10-03T20:00:23.746 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2023-10-03T20:00:23.758 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2023-10-03T20:00:23.758 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2023-10-03T20:00:23.758 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2023-10-03T20:00:23.758 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2023-10-03T20:00:23.758 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 20 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.15.0.84.81 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2023-10-03T20:00:23.759 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2023-10-03T20:00:23.760 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2023-10-03T20:00:23.807 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.ykWqUNl0Bf 2023-10-03T20:00:23.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ykWqUNl0Bf 2023-10-03T20:00:23.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.ykWqUNl0Bf 2023-10-03T20:00:23.976 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.ykWqUNl0Bf is 10KB 2023-10-03T20:00:23.990 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.ykWqUNl0Bf 2023-10-03T20:00:23.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-03T20:00:24.046 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:00:24.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-03T20:00:24.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-03T20:00:24.109 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-84-generic 2023-10-03T20:00:24.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo update-grub 2023-10-03T20:00:25.226 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub' 2023-10-03T20:00:25.228 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-03T20:00:25.231 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2023-10-03T20:00:25.486 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-10-03T20:00:25.491 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-10-03T20:00:25.796 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-03T20:00:25.801 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-03T20:00:25.841 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-03T20:00:25.847 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-03T20:00:26.004 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-03T20:00:26.005 INFO:teuthology.orchestra.run.smithi167.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-03T20:00:26.005 INFO:teuthology.orchestra.run.smithi167.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-03T20:00:26.032 INFO:teuthology.orchestra.run.smithi167.stderr:done 2023-10-03T20:00:26.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2023-10-03T20:00:56.070 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-03T20:00:56.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:00:56.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T20:01:14.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-03T20:01:23.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:01:23.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T20:01:26.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-03T20:01:38.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:01:38.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T20:01:57.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-03T20:02:12.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:02:12.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T20:02:12.133 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-03T20:02:30.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:02:30.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T20:02:30.418 DEBUG:teuthology.orchestra.run.smithi167:> true 2023-10-03T20:02:30.966 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:02:30.966 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-84-generic"... 2023-10-03T20:02:30.966 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-10-03T20:02:31.014 INFO:teuthology.orchestra.run.smithi167.stdout:5.15.0-84-generic 2023-10-03T20:02:31.014 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-84-generic vs 5.15.0-84-generic 2023-10-03T20:02:31.014 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-03T20:02:31.015 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-03T20:02:32.015 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-03T20:02:32.016 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-03T20:02:32.088 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2023-10-03T20:02:32.115 DEBUG:teuthology.parallel:result is None 2023-10-03T20:02:32.116 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-03T20:02:32.125 INFO:teuthology.task.internal:Creating test directory... 2023-10-03T20:02:32.125 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-03T20:02:32.131 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-03T20:02:32.137 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-03T20:02:32.149 INFO:teuthology.task.internal:Creating archive directory... 2023-10-03T20:02:32.149 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-03T20:02:32.192 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-03T20:02:32.200 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-03T20:02:32.200 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-03T20:02:32.252 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T20:02:32.259 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T20:02:32.261 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-03T20:02:32.269 INFO:teuthology.task.internal:Configuring sudo... 2023-10-03T20:02:32.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-03T20:02:32.317 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-03T20:02:32.326 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-03T20:02:32.326 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-03T20:02:32.366 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T20:02:32.414 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T20:02:32.463 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:02:32.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-03T20:02:32.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2023-10-03T20:02:32.628 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-03T20:02:32.636 INFO:teuthology.task.internal:Starting timer... 2023-10-03T20:02:32.636 INFO:teuthology.run_tasks:Running task pcp... 2023-10-03T20:02:32.656 INFO:teuthology.run_tasks:Running task selinux... 2023-10-03T20:02:32.664 DEBUG:teuthology.task.selinux:Excluding smithi167: OS 'ubuntu' does not support SELinux 2023-10-03T20:02:32.664 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-03T20:02:32.664 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-03T20:02:32.664 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-03T20:02:32.665 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-03T20:02:32.674 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-03T20:02:32.674 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-03T20:02:32.694 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-03T20:02:32.695 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-03T20:10:54.661 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2023-10-03T20:10:54.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:10:54.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T20:10:54.742 DEBUG:teuthology.orchestra.run.smithi167:> true 2023-10-03T20:10:54.850 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-03T20:10:54.851 INFO:teuthology.run_tasks:Running task clock... 2023-10-03T20:10:54.861 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-03T20:10:54.861 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-03T20:10:54.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T20:10:54.917 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-03T20:10:54.917 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Command line: ntpd -gq 2023-10-03T20:10:54.917 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: ---------------------------------------------------- 2023-10-03T20:10:54.917 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: ntp-4 is maintained by Network Time Foundation, 2023-10-03T20:10:54.917 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-03T20:10:54.918 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: corporation. Support and training for ntp-4 are 2023-10-03T20:10:54.918 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: available at https://www.nwtime.org/support 2023-10-03T20:10:54.918 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: ---------------------------------------------------- 2023-10-03T20:10:54.918 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: proto: precision = 0.044 usec (-24) 2023-10-03T20:10:54.918 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: basedate set to 2022-02-04 2023-10-03T20:10:54.919 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: gps base set to 2022-02-06 (week 2196) 2023-10-03T20:10:54.919 INFO:teuthology.orchestra.run.smithi167.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-03T20:10:54.919 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-03T20:10:54.919 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: restrict ::: KOD does nothing without LIMITED. 2023-10-03T20:10:54.920 INFO:teuthology.orchestra.run.smithi167.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-03T20:10:54.920 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listen and drop on 0 v6wildcard [::]:123 2023-10-03T20:10:54.920 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-03T20:10:54.921 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listen normally on 2 lo 127.0.0.1:123 2023-10-03T20:10:54.921 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listen normally on 3 enp3s0f1 172.21.15.167:123 2023-10-03T20:10:54.921 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listen normally on 4 lo [::1]:123 2023-10-03T20:10:54.921 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7309%5]:123 2023-10-03T20:10:54.921 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:10:54 ntpd[27820]: Listening on routing socket on fd #22 for interface updates 2023-10-03T20:10:55.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:55.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:55.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:55.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:55.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:55.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:55.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:10:55.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:55 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:10:56.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:56 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:56.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:56 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:56.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:56 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:56.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:56 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:56.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:56 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:10:56.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:56 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:10:57.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:57 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:57.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:57 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:57.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:57 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:57.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:57 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:57.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:57 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:10:57.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:57 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:10:58.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:10:58.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:58 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:10:59.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:59 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:59.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:59 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:59.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:59 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:10:59.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:59 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:10:59.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:59 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:10:59.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:10:59 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:11:00.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:11:00.920 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:11:00.920 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:11:00.920 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:11:00.920 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:00 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:11:01.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:11:01.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:11:01.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T20:11:01.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T20:11:01.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T20:11:01.918 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T20:11:01.919 INFO:teuthology.orchestra.run.smithi167.stdout: 3 Oct 20:11:01 ntpd[27820]: ntpd: time slew -0.003145 s 2023-10-03T20:11:01.919 INFO:teuthology.orchestra.run.smithi167.stdout:ntpd: time slew -0.003145s 2023-10-03T20:11:01.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-03T20:11:01.919 INFO:teuthology.orchestra.run.smithi167.stderr: 3 Oct 20:11:01 ntpd[27820]: can't open /var/log/ntpstats/loopstats.20231003: Permission denied 2023-10-03T20:11:02.015 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2023-10-03T20:11:02.015 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2023-10-03T20:11:02.015 INFO:teuthology.orchestra.run.smithi167.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T20:11:02.015 INFO:teuthology.orchestra.run.smithi167.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T20:11:02.016 INFO:teuthology.orchestra.run.smithi167.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T20:11:02.016 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T20:11:02.016 INFO:teuthology.run_tasks:Running task install... 2023-10-03T20:11:02.027 DEBUG:teuthology.task.install:project ceph 2023-10-03T20:11:02.027 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-03T20:11:02.027 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-03T20:11:02.028 INFO:teuthology.task.install:Using flavor: default 2023-10-03T20:11:02.036 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-03T20:11:02.036 INFO:teuthology.task.install:extra packages: [] 2023-10-03T20:11:02.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-key list | grep Ceph 2023-10-03T20:11:02.115 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-03T20:11:02.145 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-03T20:11:02.145 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph.com (release key) 2023-10-03T20:11:02.146 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-03T20:11:02.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T20:11:02.303 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default/ 2023-10-03T20:11:02.303 INFO:teuthology.task.install.deb:Package version is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:11:02.576 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:11:02.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-03T20:11:02.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-03T20:11:02.824 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T20:11:02.866 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-03T20:11:02.880 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-03T20:11:02.909 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-03T20:11:02.968 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy InRelease 2023-10-03T20:11:03.037 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-03T20:11:03.115 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-03T20:11:03.190 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.6 kB] 2023-10-03T20:11:03.349 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-03T20:11:03.436 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 28.6 kB in 1s (43.8 kB/s) 2023-10-03T20:11:04.309 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:11:04.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6336-ga7de91bf-1jammy cephadm=18.0.0-6336-ga7de91bf-1jammy ceph-mds=18.0.0-6336-ga7de91bf-1jammy ceph-mgr=18.0.0-6336-ga7de91bf-1jammy ceph-common=18.0.0-6336-ga7de91bf-1jammy ceph-fuse=18.0.0-6336-ga7de91bf-1jammy ceph-test=18.0.0-6336-ga7de91bf-1jammy ceph-volume=18.0.0-6336-ga7de91bf-1jammy radosgw=18.0.0-6336-ga7de91bf-1jammy python3-rados=18.0.0-6336-ga7de91bf-1jammy python3-rgw=18.0.0-6336-ga7de91bf-1jammy python3-cephfs=18.0.0-6336-ga7de91bf-1jammy python3-rbd=18.0.0-6336-ga7de91bf-1jammy libcephfs2=18.0.0-6336-ga7de91bf-1jammy libcephfs-dev=18.0.0-6336-ga7de91bf-1jammy librados2=18.0.0-6336-ga7de91bf-1jammy librbd1=18.0.0-6336-ga7de91bf-1jammy rbd-fuse=18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:11:04.382 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:11:04.544 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-03T20:11:04.545 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-03T20:11:04.690 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T20:11:04.690 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T20:11:04.691 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T20:11:04.691 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T20:11:04.691 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-03T20:11:04.692 INFO:teuthology.orchestra.run.smithi167.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-03T20:11:04.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-03T20:11:04.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout: python3-watchdog 2023-10-03T20:11:04.695 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2023-10-03T20:11:04.696 INFO:teuthology.orchestra.run.smithi167.stdout: btrfs-tools 2023-10-03T20:11:04.787 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2023-10-03T20:11:04.788 INFO:teuthology.orchestra.run.smithi167.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-03T20:11:04.788 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-03T20:11:04.788 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-03T20:11:04.788 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-03T20:11:04.788 INFO:teuthology.orchestra.run.smithi167.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-03T20:11:04.788 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-03T20:11:04.789 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-03T20:11:04.790 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-03T20:11:04.791 INFO:teuthology.orchestra.run.smithi167.stdout: socat xmlstarlet 2023-10-03T20:11:05.206 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 109 newly installed, 0 to remove and 254 not upgraded. 2023-10-03T20:11:05.207 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 183 MB of archives. 2023-10-03T20:11:05.207 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 823 MB of additional disk space will be used. 2023-10-03T20:11:05.207 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6336-ga7de91bf-1jammy [3,437 kB] 2023-10-03T20:11:05.975 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6336-ga7de91bf-1jammy [3,121 kB] 2023-10-03T20:11:06.312 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6336-ga7de91bf-1jammy [676 kB] 2023-10-03T20:11:06.393 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6336-ga7de91bf-1jammy [345 kB] 2023-10-03T20:11:06.440 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6336-ga7de91bf-1jammy [23.0 kB] 2023-10-03T20:11:06.442 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6336-ga7de91bf-1jammy [170 kB] 2023-10-03T20:11:06.458 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6336-ga7de91bf-1jammy [52.4 kB] 2023-10-03T20:11:06.463 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6336-ga7de91bf-1jammy [318 kB] 2023-10-03T20:11:06.510 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6336-ga7de91bf-1jammy [6,201 kB] 2023-10-03T20:11:07.249 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6336-ga7de91bf-1jammy [102 kB] 2023-10-03T20:11:07.296 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6336-ga7de91bf-1jammy [428 kB] 2023-10-03T20:11:07.327 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6336-ga7de91bf-1jammy [22.5 MB] 2023-10-03T20:11:09.452 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6336-ga7de91bf-1jammy [6,840 kB] 2023-10-03T20:11:09.832 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6336-ga7de91bf-1jammy [228 kB] 2023-10-03T20:11:09.837 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6336-ga7de91bf-1jammy [117 kB] 2023-10-03T20:11:09.839 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6336-ga7de91bf-1jammy [1,487 kB] 2023-10-03T20:11:09.948 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6336-ga7de91bf-1jammy [9,851 kB] 2023-10-03T20:11:10.854 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6336-ga7de91bf-1jammy [31.7 MB] 2023-10-03T20:11:12.412 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-03T20:11:12.470 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-03T20:11:12.507 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-03T20:11:12.509 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-03T20:11:12.524 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-03T20:11:12.578 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-03T20:11:12.582 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-03T20:11:12.595 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-03T20:11:12.600 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-03T20:11:12.600 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-03T20:11:12.600 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-03T20:11:12.608 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-03T20:11:12.609 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-03T20:11:12.609 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-03T20:11:12.618 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-03T20:11:12.640 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-03T20:11:12.641 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-03T20:11:12.641 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-03T20:11:12.642 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-03T20:11:12.642 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-03T20:11:12.642 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-03T20:11:12.645 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-03T20:11:12.646 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-03T20:11:12.646 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-03T20:11:12.667 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-03T20:11:12.668 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-03T20:11:12.668 INFO:teuthology.orchestra.run.smithi167.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-03T20:11:12.669 INFO:teuthology.orchestra.run.smithi167.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-03T20:11:12.670 INFO:teuthology.orchestra.run.smithi167.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-03T20:11:12.670 INFO:teuthology.orchestra.run.smithi167.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-03T20:11:12.684 INFO:teuthology.orchestra.run.smithi167.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-03T20:11:12.690 INFO:teuthology.orchestra.run.smithi167.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-03T20:11:12.690 INFO:teuthology.orchestra.run.smithi167.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-03T20:11:12.690 INFO:teuthology.orchestra.run.smithi167.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-03T20:11:12.701 INFO:teuthology.orchestra.run.smithi167.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-03T20:11:12.703 INFO:teuthology.orchestra.run.smithi167.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-03T20:11:12.705 INFO:teuthology.orchestra.run.smithi167.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-03T20:11:12.705 INFO:teuthology.orchestra.run.smithi167.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-03T20:11:12.707 INFO:teuthology.orchestra.run.smithi167.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-03T20:11:12.707 INFO:teuthology.orchestra.run.smithi167.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-03T20:11:12.718 INFO:teuthology.orchestra.run.smithi167.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-03T20:11:12.719 INFO:teuthology.orchestra.run.smithi167.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-03T20:11:12.721 INFO:teuthology.orchestra.run.smithi167.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-03T20:11:12.775 INFO:teuthology.orchestra.run.smithi167.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-03T20:11:12.776 INFO:teuthology.orchestra.run.smithi167.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-03T20:11:12.777 INFO:teuthology.orchestra.run.smithi167.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-03T20:11:12.778 INFO:teuthology.orchestra.run.smithi167.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-03T20:11:12.778 INFO:teuthology.orchestra.run.smithi167.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-03T20:11:12.779 INFO:teuthology.orchestra.run.smithi167.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-03T20:11:12.796 INFO:teuthology.orchestra.run.smithi167.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-03T20:11:12.798 INFO:teuthology.orchestra.run.smithi167.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-03T20:11:12.798 INFO:teuthology.orchestra.run.smithi167.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-03T20:11:12.813 INFO:teuthology.orchestra.run.smithi167.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-03T20:11:12.813 INFO:teuthology.orchestra.run.smithi167.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-03T20:11:12.813 INFO:teuthology.orchestra.run.smithi167.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-03T20:11:12.814 INFO:teuthology.orchestra.run.smithi167.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-03T20:11:12.814 INFO:teuthology.orchestra.run.smithi167.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-03T20:11:12.814 INFO:teuthology.orchestra.run.smithi167.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-03T20:11:12.827 INFO:teuthology.orchestra.run.smithi167.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-03T20:11:12.828 INFO:teuthology.orchestra.run.smithi167.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-03T20:11:12.831 INFO:teuthology.orchestra.run.smithi167.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-03T20:11:12.844 INFO:teuthology.orchestra.run.smithi167.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-03T20:11:12.845 INFO:teuthology.orchestra.run.smithi167.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-03T20:11:12.846 INFO:teuthology.orchestra.run.smithi167.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-03T20:11:12.849 INFO:teuthology.orchestra.run.smithi167.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-03T20:11:12.851 INFO:teuthology.orchestra.run.smithi167.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-03T20:11:12.855 INFO:teuthology.orchestra.run.smithi167.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-03T20:11:12.858 INFO:teuthology.orchestra.run.smithi167.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-03T20:11:12.861 INFO:teuthology.orchestra.run.smithi167.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-03T20:11:12.861 INFO:teuthology.orchestra.run.smithi167.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-03T20:11:12.862 INFO:teuthology.orchestra.run.smithi167.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-03T20:11:12.878 INFO:teuthology.orchestra.run.smithi167.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-03T20:11:12.879 INFO:teuthology.orchestra.run.smithi167.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-03T20:11:12.886 INFO:teuthology.orchestra.run.smithi167.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-03T20:11:12.886 INFO:teuthology.orchestra.run.smithi167.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-03T20:11:12.886 INFO:teuthology.orchestra.run.smithi167.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-03T20:11:12.888 INFO:teuthology.orchestra.run.smithi167.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-03T20:11:12.888 INFO:teuthology.orchestra.run.smithi167.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-03T20:11:14.621 INFO:teuthology.orchestra.run.smithi167.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6336-ga7de91bf-1jammy [3,596 B] 2023-10-03T20:11:14.622 INFO:teuthology.orchestra.run.smithi167.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6336-ga7de91bf-1jammy [827 kB] 2023-10-03T20:11:14.638 INFO:teuthology.orchestra.run.smithi167.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6336-ga7de91bf-1jammy [2,135 kB] 2023-10-03T20:11:14.897 INFO:teuthology.orchestra.run.smithi167.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6336-ga7de91bf-1jammy [251 kB] 2023-10-03T20:11:14.932 INFO:teuthology.orchestra.run.smithi167.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6336-ga7de91bf-1jammy [128 kB] 2023-10-03T20:11:14.936 INFO:teuthology.orchestra.run.smithi167.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6336-ga7de91bf-1jammy [1,883 kB] 2023-10-03T20:11:15.180 INFO:teuthology.orchestra.run.smithi167.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6336-ga7de91bf-1jammy [8,615 kB] 2023-10-03T20:11:16.298 INFO:teuthology.orchestra.run.smithi167.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6336-ga7de91bf-1jammy [3,722 B] 2023-10-03T20:11:16.318 INFO:teuthology.orchestra.run.smithi167.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6336-ga7de91bf-1jammy [47.7 MB] 2023-10-03T20:11:23.148 INFO:teuthology.orchestra.run.smithi167.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6336-ga7de91bf-1jammy [234 kB] 2023-10-03T20:11:23.152 INFO:teuthology.orchestra.run.smithi167.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6336-ga7de91bf-1jammy [28.3 kB] 2023-10-03T20:11:23.153 INFO:teuthology.orchestra.run.smithi167.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6336-ga7de91bf-1jammy [12.3 MB] 2023-10-03T20:11:25.231 INFO:teuthology.orchestra.run.smithi167.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6336-ga7de91bf-1jammy [80.9 kB] 2023-10-03T20:11:25.757 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 183 MB in 20s (8,965 kB/s) 2023-10-03T20:11:25.901 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-03T20:11:25.999 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 166009 files and directories currently installed.) 2023-10-03T20:11:26.005 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-03T20:11:26.109 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-03T20:11:26.412 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-03T20:11:26.438 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-03T20:11:26.479 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-03T20:11:26.807 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-03T20:11:26.833 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-03T20:11:26.874 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-03T20:11:27.210 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-03T20:11:27.236 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-03T20:11:27.277 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-03T20:11:27.672 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-03T20:11:27.698 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-03T20:11:27.746 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T20:11:28.284 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-03T20:11:28.310 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-03T20:11:28.350 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T20:11:28.695 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-03T20:11:28.722 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-03T20:11:28.761 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T20:11:29.090 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-03T20:11:29.117 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-03T20:11:29.148 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-03T20:11:29.434 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librados2. 2023-10-03T20:11:29.461 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../008-librados2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:29.501 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librados2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:29.971 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librbd1. 2023-10-03T20:11:29.997 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../009-librbd1_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:30.037 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librbd1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:30.465 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs2. 2023-10-03T20:11:30.492 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:30.532 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:30.784 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rados. 2023-10-03T20:11:30.807 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:30.851 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rados (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:31.154 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-03T20:11:31.180 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:11:31.220 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-argparse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:31.473 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cephfs. 2023-10-03T20:11:31.499 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:31.539 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cephfs (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:31.817 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-03T20:11:31.844 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:11:31.875 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:32.179 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-03T20:11:32.205 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-03T20:11:32.245 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-03T20:11:32.565 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-prettytable. 2023-10-03T20:11:32.591 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-03T20:11:32.632 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-03T20:11:32.901 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rbd. 2023-10-03T20:11:32.927 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:32.968 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rbd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:33.279 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-03T20:11:33.306 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-03T20:11:33.337 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-03T20:11:33.807 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librgw2. 2023-10-03T20:11:33.834 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../019-librgw2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:33.874 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librgw2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:34.401 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rgw. 2023-10-03T20:11:34.417 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:34.459 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:34.746 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-03T20:11:34.759 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-03T20:11:34.788 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-03T20:11:35.124 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libradosstriper1. 2023-10-03T20:11:35.151 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:35.190 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libradosstriper1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:35.619 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-common. 2023-10-03T20:11:35.646 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:35.686 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:37.000 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-base. 2023-10-03T20:11:37.026 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:37.075 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-base (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:37.738 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-03T20:11:37.765 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-03T20:11:37.796 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-03T20:11:39.033 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cheroot. 2023-10-03T20:11:39.059 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-03T20:11:39.099 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-03T20:11:39.436 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-03T20:11:39.463 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-03T20:11:39.502 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-03T20:11:39.797 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-03T20:11:39.824 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-03T20:11:39.863 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-03T20:11:40.116 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-03T20:11:40.143 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-03T20:11:40.183 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-03T20:11:40.519 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempora. 2023-10-03T20:11:40.546 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-03T20:11:40.586 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-03T20:11:40.889 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-portend. 2023-10-03T20:11:40.916 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-03T20:11:40.955 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-03T20:11:41.200 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-03T20:11:41.226 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-03T20:11:41.267 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-03T20:11:41.578 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-03T20:11:41.605 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-03T20:11:41.645 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-03T20:11:41.982 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-natsort. 2023-10-03T20:11:42.008 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-03T20:11:42.048 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-03T20:11:42.360 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-logutils. 2023-10-03T20:11:42.386 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-03T20:11:42.426 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-03T20:11:42.729 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-03T20:11:42.756 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-03T20:11:42.796 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-03T20:11:43.099 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-mako. 2023-10-03T20:11:43.125 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-03T20:11:43.157 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-03T20:11:43.485 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-03T20:11:43.512 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-03T20:11:43.552 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-03T20:11:43.830 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-03T20:11:43.857 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-03T20:11:43.897 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-03T20:11:44.183 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webob. 2023-10-03T20:11:44.210 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-03T20:11:44.241 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-03T20:11:44.552 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-waitress. 2023-10-03T20:11:44.579 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-03T20:11:44.632 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-03T20:11:44.941 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempita. 2023-10-03T20:11:44.968 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-03T20:11:45.007 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-03T20:11:45.310 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-paste. 2023-10-03T20:11:45.337 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-03T20:11:45.377 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-03T20:11:45.672 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-03T20:11:45.690 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-03T20:11:45.730 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-03T20:11:46.016 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-03T20:11:46.043 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-03T20:11:46.083 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-03T20:11:46.386 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webtest. 2023-10-03T20:11:46.412 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-03T20:11:46.452 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-03T20:11:46.755 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pecan. 2023-10-03T20:11:46.782 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-03T20:11:46.813 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-03T20:11:47.143 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-certifi. 2023-10-03T20:11:47.159 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-03T20:11:47.193 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-03T20:11:47.504 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-idna. 2023-10-03T20:11:47.534 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-03T20:11:47.596 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-03T20:11:47.916 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-urllib3. 2023-10-03T20:11:47.943 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-03T20:11:47.982 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-03T20:11:48.302 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests. 2023-10-03T20:11:48.329 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-03T20:11:48.368 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-03T20:11:48.680 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-03T20:11:48.707 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-03T20:11:48.746 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-03T20:11:49.199 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-03T20:11:49.226 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:11:49.266 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:49.629 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-03T20:11:49.656 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:49.695 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:50.015 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-openssl. 2023-10-03T20:11:50.042 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-03T20:11:50.081 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-03T20:11:50.385 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr. 2023-10-03T20:11:50.411 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:50.451 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:50.855 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mon. 2023-10-03T20:11:50.881 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:50.921 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mon (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:51.800 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-osd. 2023-10-03T20:11:51.826 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:51.858 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-osd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:53.702 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph. 2023-10-03T20:11:53.729 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../059-ceph_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:53.769 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:54.097 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-fuse. 2023-10-03T20:11:54.123 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:54.163 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:54.667 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mds. 2023-10-03T20:11:54.693 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:54.753 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mds (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:55.128 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephadm. 2023-10-03T20:11:55.155 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../062-cephadm_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:11:55.186 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:55.490 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-03T20:11:55.516 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-03T20:11:55.556 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-03T20:11:55.876 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-03T20:11:55.919 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-03T20:11:55.943 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-03T20:11:56.981 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-babel. 2023-10-03T20:11:57.008 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-03T20:11:57.047 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-03T20:11:57.367 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jinja2. 2023-10-03T20:11:57.393 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-03T20:11:57.433 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-03T20:11:57.803 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-03T20:11:57.830 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:11:57.870 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:11:58.207 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-03T20:11:58.234 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-03T20:11:58.273 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-03T20:11:58.601 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-03T20:11:58.628 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-03T20:11:58.668 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-03T20:11:58.912 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-routes. 2023-10-03T20:11:58.939 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-03T20:11:58.970 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-03T20:11:59.265 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-03T20:11:59.281 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:11:59.315 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:00.002 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-03T20:12:00.017 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-03T20:12:00.052 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-03T20:12:00.531 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-joblib. 2023-10-03T20:12:00.558 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-03T20:12:00.598 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-03T20:12:00.959 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-03T20:12:00.987 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-03T20:12:01.026 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-03T20:12:01.312 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn. 2023-10-03T20:12:01.339 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-03T20:12:01.370 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-03T20:12:02.079 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-03T20:12:02.107 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:12:02.146 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:03.099 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cachetools. 2023-10-03T20:12:03.127 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-03T20:12:03.165 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-03T20:12:03.443 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-03T20:12:03.471 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-03T20:12:03.501 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-03T20:12:03.830 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-03T20:12:03.857 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-03T20:12:03.896 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-03T20:12:04.216 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rsa. 2023-10-03T20:12:04.243 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-03T20:12:04.282 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-03T20:12:04.586 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-google-auth. 2023-10-03T20:12:04.613 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-03T20:12:04.652 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-03T20:12:04.931 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-03T20:12:04.958 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-03T20:12:05.142 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-03T20:12:05.569 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-websocket. 2023-10-03T20:12:05.596 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-03T20:12:05.635 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-03T20:12:05.914 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-03T20:12:05.941 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-03T20:12:05.971 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-03T20:12:06.686 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-03T20:12:06.713 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:12:06.752 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:07.072 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-03T20:12:07.099 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-03T20:12:07.138 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-03T20:12:07.441 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-03T20:12:07.469 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-03T20:12:07.508 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-03T20:12:07.744 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package jq. 2023-10-03T20:12:07.767 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-03T20:12:07.810 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-03T20:12:08.063 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package socat. 2023-10-03T20:12:08.079 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-03T20:12:08.121 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-03T20:12:08.450 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package xmlstarlet. 2023-10-03T20:12:08.477 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-03T20:12:08.516 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-03T20:12:08.871 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-test. 2023-10-03T20:12:08.898 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:12:08.937 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-test (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:11.182 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-volume. 2023-10-03T20:12:11.198 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T20:12:11.241 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-volume (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:11.529 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-03T20:12:11.556 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:12:11.587 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs-dev (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:11.940 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package nvme-cli. 2023-10-03T20:12:11.968 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-03T20:12:12.007 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-03T20:12:12.362 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-03T20:12:12.378 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-03T20:12:12.420 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-03T20:12:12.857 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-03T20:12:12.884 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-03T20:12:12.923 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-03T20:12:13.226 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastescript. 2023-10-03T20:12:13.242 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-03T20:12:13.284 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-03T20:12:13.596 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pluggy. 2023-10-03T20:12:13.624 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-03T20:12:13.662 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-03T20:12:13.965 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-psutil. 2023-10-03T20:12:13.993 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-03T20:12:14.032 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-03T20:12:14.368 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-py. 2023-10-03T20:12:14.396 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-03T20:12:14.435 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-03T20:12:14.755 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pygments. 2023-10-03T20:12:14.782 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-03T20:12:14.821 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-03T20:12:15.317 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-03T20:12:15.345 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-03T20:12:15.383 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-03T20:12:15.928 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-toml. 2023-10-03T20:12:15.955 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-03T20:12:15.994 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-03T20:12:16.297 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pytest. 2023-10-03T20:12:16.324 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-03T20:12:16.363 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-03T20:12:16.709 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplejson. 2023-10-03T20:12:16.736 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-03T20:12:16.775 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-03T20:12:17.053 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-03T20:12:17.081 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-03T20:12:17.120 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-03T20:12:17.890 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package radosgw. 2023-10-03T20:12:17.918 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../107-radosgw_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:12:17.957 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking radosgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:18.718 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package rbd-fuse. 2023-10-03T20:12:18.734 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T20:12:18.768 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking rbd-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:19.038 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-03T20:12:19.289 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-03T20:12:19.424 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-03T20:12:19.688 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-03T20:12:20.037 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-03T20:12:20.504 INFO:teuthology.orchestra.run.smithi167.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-03T20:12:20.568 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:21.095 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user cephadm....done 2023-10-03T20:12:21.241 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-03T20:12:21.517 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-03T20:12:21.785 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-03T20:12:21.886 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-03T20:12:22.120 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-03T20:12:22.355 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-03T20:12:22.481 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-03T20:12:22.783 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-03T20:12:23.151 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-03T20:12:23.425 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-03T20:12:23.687 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-03T20:12:23.913 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-03T20:12:24.164 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-03T20:12:24.423 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-argparse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:24.683 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-03T20:12:24.809 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-03T20:12:25.068 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-03T20:12:25.177 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-03T20:12:25.528 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-03T20:12:25.629 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-03T20:12:25.872 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-03T20:12:26.224 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-03T20:12:26.532 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-03T20:12:27.210 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-03T20:12:27.470 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-03T20:12:27.579 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-03T20:12:27.680 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-03T20:12:27.955 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-03T20:12:28.334 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-03T20:12:28.585 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-03T20:12:28.945 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-03T20:12:29.222 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-03T20:12:29.565 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-03T20:12:29.816 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-03T20:12:29.942 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:30.235 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-idna (3.3-1) ... 2023-10-03T20:12:30.553 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-03T20:12:31.742 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T20:12:31.876 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-03T20:12:32.119 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-03T20:12:32.229 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-03T20:12:32.330 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-03T20:12:32.614 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-03T20:12:32.866 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-03T20:12:33.142 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T20:12:33.276 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-03T20:12:33.402 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-03T20:12:33.704 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-03T20:12:33.946 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-03T20:12:34.206 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-03T20:12:34.457 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-03T20:12:34.784 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-03T20:12:35.043 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-03T20:12:35.169 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-03T20:12:35.437 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-03T20:12:35.563 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-03T20:12:35.864 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-03T20:12:36.165 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-03T20:12:36.416 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-03T20:12:36.858 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-03T20:12:37.094 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-03T20:12:37.212 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-03T20:12:37.605 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-03T20:12:37.865 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-03T20:12:38.116 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T20:12:38.251 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-03T20:12:38.560 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-03T20:12:38.853 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-03T20:12:39.112 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-03T20:12:39.222 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-03T20:12:39.481 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-03T20:12:39.749 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-03T20:12:40.017 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-03T20:12:40.393 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-03T20:12:40.670 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-03T20:12:41.022 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-03T20:12:41.148 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librados2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:41.274 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librgw2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:41.434 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:41.569 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:41.686 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libradosstriper1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:41.889 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-03T20:12:42.166 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librbd1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:42.300 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-modules-core (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:42.426 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:42.639 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-03T20:12:42.639 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-03T20:12:43.145 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs-dev (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:43.296 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rados (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:43.414 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-03T20:12:44.661 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rbd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:44.779 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up rbd-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:44.923 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:45.047 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cephfs (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:45.173 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:45.700 INFO:teuthology.orchestra.run.smithi167.stdout:Adding group ceph....done 2023-10-03T20:12:46.068 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user ceph....done 2023-10-03T20:12:46.237 INFO:teuthology.orchestra.run.smithi167.stdout:Setting system user ceph properties....done 2023-10-03T20:12:46.255 INFO:teuthology.orchestra.run.smithi167.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-03T20:12:46.390 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-03T20:12:46.764 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-03T20:12:47.317 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-test (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:47.443 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up radosgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:47.689 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-03T20:12:47.689 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-03T20:12:48.428 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-base (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:48.768 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-03T20:12:49.272 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mds (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:49.506 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-03T20:12:49.507 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-03T20:12:50.015 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:50.233 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-03T20:12:50.233 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-03T20:12:50.758 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-osd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:51.052 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-03T20:12:51.052 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-03T20:12:51.669 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:51.898 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:52.049 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mon (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:52.286 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-03T20:12:52.286 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-03T20:12:52.767 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:52.918 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:53.027 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-dashboard (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:53.162 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-volume (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T20:12:53.289 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-03T20:12:54.407 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-03T20:12:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-03T20:12:54.872 DEBUG:teuthology.parallel:result is None 2023-10-03T20:12:54.873 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T20:12:55.107 DEBUG:teuthology.orchestra.run.smithi167:> dpkg-query -W -f '${Version}' ceph 2023-10-03T20:12:55.125 INFO:teuthology.orchestra.run.smithi167.stdout:18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:12:55.126 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:12:55.126 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336-ga7de91bf-1jammy is installed. 2023-10-03T20:12:55.126 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-03T20:12:55.126 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:55.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-03T20:12:55.178 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-03T20:12:55.179 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:55.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2023-10-03T20:12:55.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-03T20:12:55.285 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-03T20:12:55.285 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:55.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-03T20:12:55.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-03T20:12:55.390 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-03T20:12:55.390 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:55.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2023-10-03T20:12:55.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-03T20:12:55.498 INFO:teuthology.run_tasks:Running task ceph... 2023-10-03T20:12:55.568 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-03T20:12:55.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2023-10-03T20:12:55.580 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-03T20:12:55.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-03T20:12:55.634 INFO:tasks.ceph:Creating extra log directories... 2023-10-03T20:12:55.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-03T20:12:55.699 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-03T20:12:55.699 INFO:tasks.ceph:config {'conf': {'client': {'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-03T20:12:55.699 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828', 'branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7409828', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-ajarr-63009-only-reacquire-kickstart'}, 'ceph': {'conf': {'client': {'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ajarr-check-addr-non-zero', 'sha1': '7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6885, 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-ajarr-check-addr-non-zero', 'suite_path': '/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ajarr/ceph', 'suite_sha1': '7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231', 'targets': {'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5dENvZ8CjFh1oP9HEOOvUxtCMq4Isbx8hO++REGaHaB5KJwIPB64PWkzDAYeBogfIJrNMGYZDotx2PmdKyH+w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-03_19:52:41', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-03T20:12:55.700 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-03T20:12:55.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2023-10-03T20:12:55.795 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:55.795 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2023-10-03T20:12:55.844 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-03T20:12:55.844 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2023-10-03T20:12:55.892 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-03T20:12:55.892 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T20:12:55.892 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 880 Links: 1 2023-10-03T20:12:55.892 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T20:12:55.893 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-03 20:07:33.248793753 +0000 2023-10-03T20:12:55.893 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-03 20:07:32.864797162 +0000 2023-10-03T20:12:55.893 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-03 20:07:32.864797162 +0000 2023-10-03T20:12:55.893 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-03T20:12:55.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-03T20:12:55.946 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-03T20:12:55.946 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-03T20:12:55.946 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000443786 s, 1.2 MB/s 2023-10-03T20:12:55.947 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-03T20:12:55.997 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2023-10-03T20:12:56.047 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-03T20:12:56.047 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T20:12:56.048 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 893 Links: 1 2023-10-03T20:12:56.048 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T20:12:56.048 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-03 20:07:33.884788107 +0000 2023-10-03T20:12:56.048 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-03 20:07:33.472791765 +0000 2023-10-03T20:12:56.048 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-03 20:07:33.472791765 +0000 2023-10-03T20:12:56.048 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-03T20:12:56.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-03T20:12:56.102 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-03T20:12:56.102 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-03T20:12:56.102 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000352118 s, 1.5 MB/s 2023-10-03T20:12:56.104 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-03T20:12:56.153 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2023-10-03T20:12:56.199 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-03T20:12:56.199 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T20:12:56.199 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 909 Links: 1 2023-10-03T20:12:56.200 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T20:12:56.200 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-03 20:07:34.520782462 +0000 2023-10-03T20:12:56.200 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-03 20:07:34.104786154 +0000 2023-10-03T20:12:56.200 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-03 20:07:34.104786154 +0000 2023-10-03T20:12:56.200 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-03T20:12:56.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-03T20:12:56.253 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-03T20:12:56.254 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-03T20:12:56.254 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000358757 s, 1.4 MB/s 2023-10-03T20:12:56.255 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-03T20:12:56.304 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2023-10-03T20:12:56.354 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-03T20:12:56.354 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T20:12:56.354 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 930 Links: 1 2023-10-03T20:12:56.354 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T20:12:56.354 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-03 20:07:35.164776743 +0000 2023-10-03T20:12:56.354 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-03 20:07:34.748780437 +0000 2023-10-03T20:12:56.355 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-03 20:07:34.748780437 +0000 2023-10-03T20:12:56.355 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-03T20:12:56.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-03T20:12:56.406 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-03T20:12:56.406 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-03T20:12:56.406 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000419424 s, 1.2 MB/s 2023-10-03T20:12:56.408 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-03T20:12:56.456 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T20:12:56.457 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-03T20:12:56.457 INFO:tasks.ceph:Generating config... 2023-10-03T20:12:56.458 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-03T20:12:56.458 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-03T20:12:56.458 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-03T20:12:56.458 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-03T20:12:56.458 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-03T20:12:56.458 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-03T20:12:56.458 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-03T20:12:56.458 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-03T20:12:56.459 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-03T20:12:56.459 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-03T20:12:56.459 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-03T20:12:56.460 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-03T20:12:56.460 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-03T20:12:56.460 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-03T20:12:56.460 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-03T20:12:56.460 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-03T20:12:56.460 INFO:tasks.ceph:Setting up mon.a... 2023-10-03T20:12:56.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-03T20:12:56.530 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.keyring 2023-10-03T20:12:56.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-03T20:12:56.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-03T20:12:56.674 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.167')] 2023-10-03T20:12:56.675 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.167', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default features': 1}, 'mon.a': {}} 2023-10-03T20:12:56.675 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:56.676 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-03T20:12:56.723 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.167 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:12:56.801 INFO:teuthology.orchestra.run.smithi167.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:12:56.802 INFO:teuthology.orchestra.run.smithi167.stdout:monmaptool: generated fsid 0eaba888-2cf8-416c-91f9-3b34fc849c21 2023-10-03T20:12:56.802 INFO:teuthology.orchestra.run.smithi167.stdout:setting min_mon_release = pacific 2023-10-03T20:12:56.802 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 0 2023-10-03T20:12:56.802 INFO:teuthology.orchestra.run.smithi167.stdout:fsid 0eaba888-2cf8-416c-91f9-3b34fc849c21 2023-10-03T20:12:56.802 INFO:teuthology.orchestra.run.smithi167.stdout:last_changed 2023-10-03T20:12:56.800931+0000 2023-10-03T20:12:56.803 INFO:teuthology.orchestra.run.smithi167.stdout:created 2023-10-03T20:12:56.800931+0000 2023-10-03T20:12:56.803 INFO:teuthology.orchestra.run.smithi167.stdout:min_mon_release 16 (pacific) 2023-10-03T20:12:56.803 INFO:teuthology.orchestra.run.smithi167.stdout:election_strategy: 1 2023-10-03T20:12:56.803 INFO:teuthology.orchestra.run.smithi167.stdout:0: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.a 2023-10-03T20:12:56.803 INFO:teuthology.orchestra.run.smithi167.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-03T20:12:56.805 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-03T20:12:56.852 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0eaba888-2cf8-416c-91f9-3b34fc849c21... 2023-10-03T20:12:56.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-03T20:12:56.927 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2023-10-03T20:12:56.927 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2023-10-03T20:12:56.927 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-03T20:12:56.927 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2023-10-03T20:12:56.928 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.929 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-03T20:12:56.930 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2023-10-03T20:12:56.931 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = 172.21.15.167 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: mon client directed command retry = 5 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on pool no app = False 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: ms inject socket failures = 5000 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 0eaba888-2cf8-416c-91f9-3b34fc849c21 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2023-10-03T20:12:56.932 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2023-10-03T20:12:56.933 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.934 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: bluefs allocator = hybrid 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore allocator = hybrid 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2023-10-03T20:12:56.935 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1/20 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1/20 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 4/10 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: osd shutdown pgref assert = True 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.936 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug rbd = 20 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2023-10-03T20:12:56.937 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2023-10-03T20:12:56.938 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.939 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout: rbd default features = 1 2023-10-03T20:12:56.940 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2023-10-03T20:12:56.941 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-03T20:12:56.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-03T20:12:57.042 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-03T20:12:57.042 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:57.042 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-03T20:12:57.048 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:57.048 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-03T20:12:57.095 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T20:12:57.095 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:57.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-03T20:12:57.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-03T20:12:57.164 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:12:57.164 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:12:57.211 INFO:tasks.ceph:Setting up mon nodes... 2023-10-03T20:12:57.211 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-03T20:12:57.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-03T20:12:57.296 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-03T20:12:57.299 INFO:tasks.ceph:Setting up mds nodes... 2023-10-03T20:12:57.299 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-03T20:12:57.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-03T20:12:57.373 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-03T20:12:57.384 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-03T20:12:57.384 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-03T20:12:57.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-03T20:12:57.447 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T20:12:57.447 INFO:tasks.ceph:role: osd.0 2023-10-03T20:12:57.447 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T20:12:57.447 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-03T20:12:57.513 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T20:12:57.513 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T20:12:57.513 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T20:12:57.514 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T20:12:57.996 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2023-10-03T20:12:57.997 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2023-10-03T20:12:57.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-03T20:12:58.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-03T20:12:58.154 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2023-10-03T20:12:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T20:12:58.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-03T20:12:58.211 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T20:12:58.211 INFO:tasks.ceph:role: osd.1 2023-10-03T20:12:58.211 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T20:12:58.211 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-03T20:12:58.274 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T20:12:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T20:12:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T20:12:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T20:12:58.274 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T20:12:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2023-10-03T20:12:58.275 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T20:12:58.275 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T20:12:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T20:12:58.275 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T20:12:58.728 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2023-10-03T20:12:58.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2023-10-03T20:12:58.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-03T20:12:58.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-03T20:12:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T20:12:58.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2023-10-03T20:12:58.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-03T20:12:58.891 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T20:12:58.891 INFO:tasks.ceph:role: osd.2 2023-10-03T20:12:58.891 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T20:12:58.891 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-03T20:12:58.955 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T20:12:58.955 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T20:12:58.955 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T20:12:58.955 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T20:12:58.955 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T20:12:58.955 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2023-10-03T20:12:58.956 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T20:12:58.956 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T20:12:58.956 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T20:12:58.956 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T20:12:59.416 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2023-10-03T20:12:59.419 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2023-10-03T20:12:59.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-03T20:12:59.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-03T20:12:59.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2023-10-03T20:12:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T20:12:59.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:12:59.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:12:59.682+0000 7f1c2371f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-03T20:12:59.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:12:59.682+0000 7f1c2371f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-03T20:12:59.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:12:59.682+0000 7f1c2371f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T20:12:59.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:12:59.682+0000 7f1c2371f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T20:12:59.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:12:59.682+0000 7f1c2371f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T20:12:59.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:12:59.690+0000 7f1c2371f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-03T20:13:02.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-03T20:13:02.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:13:02.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:02.826+0000 7f42921c2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-03T20:13:02.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:02.826+0000 7f42921c2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-03T20:13:02.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:02.826+0000 7f42921c2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T20:13:02.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:02.826+0000 7f42921c2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T20:13:02.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:02.826+0000 7f42921c2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T20:13:02.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:02.826+0000 7f42921c2640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-03T20:13:05.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-03T20:13:05.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:13:05.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:05.986+0000 7efce8d19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-03T20:13:05.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:05.986+0000 7efce8d19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-03T20:13:05.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:05.986+0000 7efce8d19640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T20:13:05.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:05.986+0000 7efce8d19640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T20:13:05.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:05.986+0000 7efce8d19640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T20:13:05.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-03T20:13:05.986+0000 7efce8d19640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-03T20:13:08.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-03T20:13:09.051 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-03T20:13:09.051 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:09.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-03T20:13:09.107 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:09.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-03T20:13:09.162 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:09.162 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-03T20:13:09.213 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:09.214 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-03T20:13:09.265 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:09.266 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-03T20:13:09.311 INFO:tasks.ceph:Adding keys to all mons... 2023-10-03T20:13:09.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.x] 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDJdRxlTXKbERAAbVU2PIP+sBN5aQ3vT425nw== 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDLdRxlljv/KBAAovbfxzW1RzFQJhvvkwTvAg== 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDOdRxleAWDMRAAlRVCdTaKKusliso6o6bsHA== 2023-10-03T20:13:09.365 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2023-10-03T20:13:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDRdRxlAsEUOxAAVzcMBzCQDlknyXI8nATTaA== 2023-10-03T20:13:09.366 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2023-10-03T20:13:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDJdRxlcH8yFhAArUnzHixOvnrV/gYXoF3B/Q== 2023-10-03T20:13:09.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-03T20:13:09.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T20:13:09.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T20:13:09.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T20:13:09.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-03T20:13:09.751 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-03T20:13:09.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-03T20:13:09.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-03T20:13:09.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-03T20:13:09.866 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-03T20:13:09.910 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-03T20:13:09.910 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-03T20:13:09.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-03T20:13:09.956 INFO:tasks.ceph.mon.a:Started 2023-10-03T20:13:09.956 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-03T20:13:09.956 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-03T20:13:09.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-03T20:13:09.958 INFO:tasks.ceph.mgr.x:Started 2023-10-03T20:13:09.959 INFO:tasks.ceph:Setting crush tunables to default 2023-10-03T20:13:09.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd crush tunables default 2023-10-03T20:13:10.203 INFO:teuthology.orchestra.run.smithi167.stderr:adjusted tunables profile to default 2023-10-03T20:13:10.221 INFO:tasks.ceph:check_enable_crimson: False 2023-10-03T20:13:10.221 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-03T20:13:10.222 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:10.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-03T20:13:10.232 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:10.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-03T20:13:10.285 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:10.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-03T20:13:10.318 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:10.314+0000 7f6f32b49180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-03T20:13:10.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 1b76dced-3877-4565-b45a-3d1abd7b031c 0 2023-10-03T20:13:10.605 INFO:teuthology.orchestra.run.smithi167.stdout:0 2023-10-03T20:13:10.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 6404d2c7-ed12-4216-a92f-68b44d9d31c5 1 2023-10-03T20:13:10.822 INFO:teuthology.orchestra.run.smithi167.stdout:1 2023-10-03T20:13:10.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new d8c3c345-38df-45e1-867c-559a9e29a2e9 2 2023-10-03T20:13:10.905 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:10.898+0000 7f6f32b49180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-03T20:13:11.032 INFO:teuthology.orchestra.run.smithi167.stdout:2 2023-10-03T20:13:11.048 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-03T20:13:11.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-03T20:13:11.050 INFO:tasks.ceph.osd.0:Started 2023-10-03T20:13:11.050 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-03T20:13:11.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-03T20:13:11.053 INFO:tasks.ceph.osd.1:Started 2023-10-03T20:13:11.053 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-03T20:13:11.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-03T20:13:11.055 INFO:tasks.ceph.osd.2:Started 2023-10-03T20:13:11.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T20:13:11.133 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:11.126+0000 7f6f32b49180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-03T20:13:11.253 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:11.253 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":5,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T20:13:11.030656+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T20:13:11.269 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-03T20:13:11.269 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-03T20:13:11.571 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T20:13:11.660 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:13:11.654+0000 7f27cabb5640 -1 Falling back to public interface 2023-10-03T20:13:11.660 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-03T20:13:11.654+0000 7f2e07a27640 -1 Falling back to public interface 2023-10-03T20:13:11.663 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:13:11.658+0000 7fb3eeaba640 -1 Falling back to public interface 2023-10-03T20:13:11.710 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:11.702+0000 7f6f32b49180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-03T20:13:11.743 INFO:teuthology.misc.health.smithi167.stdout: 2023-10-03T20:13:11.744 INFO:teuthology.misc.health.smithi167.stdout:{"epoch":5,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T20:13:11.030656+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T20:13:11.758 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-03T20:13:11.767 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:11.762+0000 7f6f32b49180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-03T20:13:11.819 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:11.814+0000 7f6f32b49180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-03T20:13:11.896 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:11.890+0000 7f6f32b49180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-03T20:13:11.947 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:11.942+0000 7f6f32b49180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-03T20:13:12.025 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:12.018+0000 7f6f32b49180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-03T20:13:12.076 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:12.070+0000 7f6f32b49180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-03T20:13:12.215 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:12.210+0000 7f6f32b49180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-03T20:13:12.274 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:12.266+0000 7f6f32b49180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-03T20:13:12.935 INFO:tasks.ceph.mgr.x.smithi167.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-03T20:13:12.935 INFO:tasks.ceph.mgr.x.smithi167.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-03T20:13:12.935 INFO:tasks.ceph.mgr.x.smithi167.stderr: from numpy import show_config as show_numpy_config 2023-10-03T20:13:12.981 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:12.974+0000 7f6f32b49180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-03T20:13:13.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:13.106+0000 7f6f32b49180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-03T20:13:13.272 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:13.266+0000 7f6f32b49180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-03T20:13:13.335 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:13.330+0000 7f6f32b49180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-03T20:13:13.456 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:13.450+0000 7f6f32b49180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-03T20:13:14.037 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:14.030+0000 7f6f32b49180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-03T20:13:14.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:14.106+0000 7f6f32b49180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-03T20:13:14.286 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:14.278+0000 7f6f32b49180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-03T20:13:14.358 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:14.350+0000 7f6f32b49180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-03T20:13:14.661 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:14.654+0000 7f6f32b49180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-03T20:13:14.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:14.714+0000 7f6f32b49180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-03T20:13:15.309 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-03T20:13:15.302+0000 7f2e07a27640 -1 osd.2 0 log_to_monitors true 2023-10-03T20:13:15.543 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:13:15.538+0000 7fb3eeaba640 -1 osd.0 0 log_to_monitors true 2023-10-03T20:13:15.610 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:13:15.602+0000 7f27cabb5640 -1 osd.1 0 log_to_monitors true 2023-10-03T20:13:15.755 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:13:15.750+0000 7f6eda2ba640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-03T20:13:18.061 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T20:13:18.496 INFO:teuthology.misc.health.smithi167.stdout: 2023-10-03T20:13:18.496 INFO:teuthology.misc.health.smithi167.stdout:{"epoch":7,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T20:13:17.040874+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T20:13:18.522 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-03T20:13:18.984 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-03T20:13:18.978+0000 7f2e039cf640 -1 osd.2 0 waiting for initial osdmap 2023-10-03T20:13:19.010 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-03T20:13:19.006+0000 7f2dfb804640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T20:13:19.014 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:13:19.010+0000 7fb3eaa62640 -1 osd.0 0 waiting for initial osdmap 2023-10-03T20:13:19.035 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:13:19.030+0000 7f27c6b5d640 -1 osd.1 0 waiting for initial osdmap 2023-10-03T20:13:19.050 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:13:19.046+0000 7fb3e2897640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T20:13:19.095 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:13:19.090+0000 7f27be992640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T20:13:24.824 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T20:13:25.113 INFO:teuthology.misc.health.smithi167.stdout: 2023-10-03T20:13:25.113 INFO:teuthology.misc.health.smithi167.stdout:{"epoch":11,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T20:13:23.092924+0000","last_up_change":"2023-10-03T20:13:20.015790+0000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T20:13:20.822429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6817","nonce":3168964859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6819","nonce":3168964859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6823","nonce":3168964859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6821","nonce":3168964859}]},"public_addr":"172.21.15.167:6817/3168964859","cluster_addr":"172.21.15.167:6819/3168964859","heartbeat_back_addr":"172.21.15.167:6823/3168964859","heartbeat_front_addr":"172.21.15.167:6821/3168964859","state":["exists","up"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6801","nonce":1617383148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6803","nonce":1617383148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6813","nonce":1617383148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6808","nonce":1617383148}]},"public_addr":"172.21.15.167:6801/1617383148","cluster_addr":"172.21.15.167:6803/1617383148","heartbeat_back_addr":"172.21.15.167:6813/1617383148","heartbeat_front_addr":"172.21.15.167:6808/1617383148","state":["exists","up"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6805","nonce":705294285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6809","nonce":705294285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6815","nonce":705294285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6812","nonce":705294285}]},"public_addr":"172.21.15.167:6805/705294285","cluster_addr":"172.21.15.167:6809/705294285","heartbeat_back_addr":"172.21.15.167:6815/705294285","heartbeat_front_addr":"172.21.15.167:6812/705294285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.550984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.577924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.350491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T20:13:25.131 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-03T20:13:25.131 INFO:tasks.ceph:Creating RBD pool 2023-10-03T20:13:25.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-03T20:13:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'rbd' created 2023-10-03T20:13:26.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-03T20:13:27.114 INFO:teuthology.orchestra.run.smithi167.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-03T20:13:27.144 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-03T20:13:27.144 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-03T20:13:27.144 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-03T20:13:27.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-03T20:13:27.145 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-03T20:13:27.470 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:27.487 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":2908110612},{"type":"v1","addr":"172.21.15.167:6825","nonce":2908110612}]},"active_addr":"172.21.15.167:6825/2908110612","active_change":"2023-10-03T20:13:14.739430+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1201200376}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1652750372}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":265975161}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":2298477283}]}]} 2023-10-03T20:13:27.489 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-03T20:13:27.489 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-03T20:13:27.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T20:13:27.763 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:27.764 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":13,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T20:13:27.111328+0000","last_up_change":"2023-10-03T20:13:20.015790+0000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T20:13:20.822429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T20:13:25.447616+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6817","nonce":3168964859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6819","nonce":3168964859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6823","nonce":3168964859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6821","nonce":3168964859}]},"public_addr":"172.21.15.167:6817/3168964859","cluster_addr":"172.21.15.167:6819/3168964859","heartbeat_back_addr":"172.21.15.167:6823/3168964859","heartbeat_front_addr":"172.21.15.167:6821/3168964859","state":["exists","up"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6801","nonce":1617383148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6803","nonce":1617383148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6813","nonce":1617383148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6808","nonce":1617383148}]},"public_addr":"172.21.15.167:6801/1617383148","cluster_addr":"172.21.15.167:6803/1617383148","heartbeat_back_addr":"172.21.15.167:6813/1617383148","heartbeat_front_addr":"172.21.15.167:6808/1617383148","state":["exists","up"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6805","nonce":705294285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6809","nonce":705294285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6815","nonce":705294285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6812","nonce":705294285}]},"public_addr":"172.21.15.167:6805/705294285","cluster_addr":"172.21.15.167:6809/705294285","heartbeat_back_addr":"172.21.15.167:6815/705294285","heartbeat_front_addr":"172.21.15.167:6812/705294285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.550984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.577924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.350491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T20:13:27.780 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-03T20:13:27.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T20:13:28.050 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:28.050 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":13,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T20:13:27.111328+0000","last_up_change":"2023-10-03T20:13:20.015790+0000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T20:13:20.822429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T20:13:25.447616+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6817","nonce":3168964859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6819","nonce":3168964859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6823","nonce":3168964859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6821","nonce":3168964859}]},"public_addr":"172.21.15.167:6817/3168964859","cluster_addr":"172.21.15.167:6819/3168964859","heartbeat_back_addr":"172.21.15.167:6823/3168964859","heartbeat_front_addr":"172.21.15.167:6821/3168964859","state":["exists","up"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6801","nonce":1617383148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6803","nonce":1617383148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6813","nonce":1617383148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6808","nonce":1617383148}]},"public_addr":"172.21.15.167:6801/1617383148","cluster_addr":"172.21.15.167:6803/1617383148","heartbeat_back_addr":"172.21.15.167:6813/1617383148","heartbeat_front_addr":"172.21.15.167:6808/1617383148","state":["exists","up"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6805","nonce":705294285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6809","nonce":705294285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6815","nonce":705294285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6812","nonce":705294285}]},"public_addr":"172.21.15.167:6805/705294285","cluster_addr":"172.21.15.167:6809/705294285","heartbeat_back_addr":"172.21.15.167:6815/705294285","heartbeat_front_addr":"172.21.15.167:6812/705294285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.550984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.577924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.350491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T20:13:28.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-03T20:13:28.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-03T20:13:28.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-03T20:13:28.218 INFO:teuthology.orchestra.run.smithi167.stdout:34359738373 2023-10-03T20:13:28.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T20:13:28.225 INFO:teuthology.orchestra.run.smithi167.stdout:34359738373 2023-10-03T20:13:28.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T20:13:28.226 INFO:teuthology.orchestra.run.smithi167.stdout:34359738372 2023-10-03T20:13:28.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T20:13:28.528 INFO:teuthology.orchestra.run.smithi167.stdout:34359738372 2023-10-03T20:13:28.541 INFO:teuthology.orchestra.run.smithi167.stdout:34359738372 2023-10-03T20:13:28.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-03T20:13:28.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-03T20:13:28.569 INFO:teuthology.orchestra.run.smithi167.stdout:34359738371 2023-10-03T20:13:28.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-10-03T20:13:29.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T20:13:29.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T20:13:29.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T20:13:29.860 INFO:teuthology.orchestra.run.smithi167.stdout:34359738373 2023-10-03T20:13:29.867 INFO:teuthology.orchestra.run.smithi167.stdout:34359738372 2023-10-03T20:13:29.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-03T20:13:29.882 DEBUG:teuthology.parallel:result is None 2023-10-03T20:13:29.888 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-10-03T20:13:29.888 DEBUG:teuthology.parallel:result is None 2023-10-03T20:13:29.892 INFO:teuthology.orchestra.run.smithi167.stdout:34359738373 2023-10-03T20:13:29.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-03T20:13:29.909 DEBUG:teuthology.parallel:result is None 2023-10-03T20:13:29.910 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-03T20:13:29.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T20:13:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:30.208 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T20:13:30.224 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-03T20:13:28.746505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.633896"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.133536+0000","last_change":"2023-10-03T20:13:27.129355+0000","last_active":"2023-10-03T20:13:27.133536+0000","last_peered":"2023-10-03T20:13:27.133536+0000","last_clean":"2023-10-03T20:13:27.133536+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T20:13:27.133536+0000","last_undegraded":"2023-10-03T20:13:27.133536+0000","last_fullsized":"2023-10-03T20:13:27.133536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:28:44.067755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.133496+0000","last_change":"2023-10-03T20:13:27.129177+0000","last_active":"2023-10-03T20:13:27.133496+0000","last_peered":"2023-10-03T20:13:27.133496+0000","last_clean":"2023-10-03T20:13:27.133496+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T20:13:27.133496+0000","last_undegraded":"2023-10-03T20:13:27.133496+0000","last_fullsized":"2023-10-03T20:13:27.133496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:12:33.146599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.133592+0000","last_change":"2023-10-03T20:13:27.128919+0000","last_active":"2023-10-03T20:13:27.133592+0000","last_peered":"2023-10-03T20:13:27.133592+0000","last_clean":"2023-10-03T20:13:27.133592+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T20:13:27.133592+0000","last_undegraded":"2023-10-03T20:13:27.133592+0000","last_fullsized":"2023-10-03T20:13:27.133592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:25:02.985161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.136199+0000","last_change":"2023-10-03T20:13:27.132141+0000","last_active":"2023-10-03T20:13:27.136199+0000","last_peered":"2023-10-03T20:13:27.136199+0000","last_clean":"2023-10-03T20:13:27.136199+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T20:13:27.136199+0000","last_undegraded":"2023-10-03T20:13:27.136199+0000","last_fullsized":"2023-10-03T20:13:27.136199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:03:48.772648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.131041+0000","last_change":"2023-10-03T20:13:27.128137+0000","last_active":"2023-10-03T20:13:27.131041+0000","last_peered":"2023-10-03T20:13:27.131041+0000","last_clean":"2023-10-03T20:13:27.131041+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T20:13:27.131041+0000","last_undegraded":"2023-10-03T20:13:27.131041+0000","last_fullsized":"2023-10-03T20:13:27.131041+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:41:15.407159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T20:13:28.128787+0000","last_change":"2023-10-03T20:13:27.130377+0000","last_active":"2023-10-03T20:13:28.128787+0000","last_peered":"2023-10-03T20:13:28.128787+0000","last_clean":"2023-10-03T20:13:28.128787+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T20:13:28.128787+0000","last_undegraded":"2023-10-03T20:13:28.128787+0000","last_fullsized":"2023-10-03T20:13:28.128787+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:18:27.043875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T20:13:28.128788+0000","last_change":"2023-10-03T20:13:27.130841+0000","last_active":"2023-10-03T20:13:28.128788+0000","last_peered":"2023-10-03T20:13:28.128788+0000","last_clean":"2023-10-03T20:13:28.128788+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T20:13:28.128788+0000","last_undegraded":"2023-10-03T20:13:28.128788+0000","last_fullsized":"2023-10-03T20:13:28.128788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:19:49.918412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.136218+0000","last_change":"2023-10-03T20:13:27.132221+0000","last_active":"2023-10-03T20:13:27.136218+0000","last_peered":"2023-10-03T20:13:27.136218+0000","last_clean":"2023-10-03T20:13:27.136218+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T20:13:27.136218+0000","last_undegraded":"2023-10-03T20:13:27.136218+0000","last_fullsized":"2023-10-03T20:13:27.136218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:52:39.347057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-03T20:13:26.121641+0000","last_change":"2023-10-03T20:13:22.539837+0000","last_active":"2023-10-03T20:13:26.121641+0000","last_peered":"2023-10-03T20:13:26.121641+0000","last_clean":"2023-10-03T20:13:26.121641+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T20:13:26.121641+0000","last_undegraded":"2023-10-03T20:13:26.121641+0000","last_fullsized":"2023-10-03T20:13:26.121641+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:15:59.949043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T20:13:30.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T20:13:30.488 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:30.489 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T20:13:30.505 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-03T20:13:28.746505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.633896"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.133536+0000","last_change":"2023-10-03T20:13:27.129355+0000","last_active":"2023-10-03T20:13:27.133536+0000","last_peered":"2023-10-03T20:13:27.133536+0000","last_clean":"2023-10-03T20:13:27.133536+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T20:13:27.133536+0000","last_undegraded":"2023-10-03T20:13:27.133536+0000","last_fullsized":"2023-10-03T20:13:27.133536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:28:44.067755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.133496+0000","last_change":"2023-10-03T20:13:27.129177+0000","last_active":"2023-10-03T20:13:27.133496+0000","last_peered":"2023-10-03T20:13:27.133496+0000","last_clean":"2023-10-03T20:13:27.133496+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T20:13:27.133496+0000","last_undegraded":"2023-10-03T20:13:27.133496+0000","last_fullsized":"2023-10-03T20:13:27.133496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:12:33.146599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.133592+0000","last_change":"2023-10-03T20:13:27.128919+0000","last_active":"2023-10-03T20:13:27.133592+0000","last_peered":"2023-10-03T20:13:27.133592+0000","last_clean":"2023-10-03T20:13:27.133592+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T20:13:27.133592+0000","last_undegraded":"2023-10-03T20:13:27.133592+0000","last_fullsized":"2023-10-03T20:13:27.133592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:25:02.985161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.136199+0000","last_change":"2023-10-03T20:13:27.132141+0000","last_active":"2023-10-03T20:13:27.136199+0000","last_peered":"2023-10-03T20:13:27.136199+0000","last_clean":"2023-10-03T20:13:27.136199+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T20:13:27.136199+0000","last_undegraded":"2023-10-03T20:13:27.136199+0000","last_fullsized":"2023-10-03T20:13:27.136199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:03:48.772648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.131041+0000","last_change":"2023-10-03T20:13:27.128137+0000","last_active":"2023-10-03T20:13:27.131041+0000","last_peered":"2023-10-03T20:13:27.131041+0000","last_clean":"2023-10-03T20:13:27.131041+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T20:13:27.131041+0000","last_undegraded":"2023-10-03T20:13:27.131041+0000","last_fullsized":"2023-10-03T20:13:27.131041+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:41:15.407159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T20:13:28.128787+0000","last_change":"2023-10-03T20:13:27.130377+0000","last_active":"2023-10-03T20:13:28.128787+0000","last_peered":"2023-10-03T20:13:28.128787+0000","last_clean":"2023-10-03T20:13:28.128787+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T20:13:28.128787+0000","last_undegraded":"2023-10-03T20:13:28.128787+0000","last_fullsized":"2023-10-03T20:13:28.128787+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:18:27.043875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T20:13:28.128788+0000","last_change":"2023-10-03T20:13:27.130841+0000","last_active":"2023-10-03T20:13:28.128788+0000","last_peered":"2023-10-03T20:13:28.128788+0000","last_clean":"2023-10-03T20:13:28.128788+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T20:13:28.128788+0000","last_undegraded":"2023-10-03T20:13:28.128788+0000","last_fullsized":"2023-10-03T20:13:28.128788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:19:49.918412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T20:13:27.136218+0000","last_change":"2023-10-03T20:13:27.132221+0000","last_active":"2023-10-03T20:13:27.136218+0000","last_peered":"2023-10-03T20:13:27.136218+0000","last_clean":"2023-10-03T20:13:27.136218+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T20:13:27.136218+0000","last_undegraded":"2023-10-03T20:13:27.136218+0000","last_fullsized":"2023-10-03T20:13:27.136218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:52:39.347057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-03T20:13:26.121641+0000","last_change":"2023-10-03T20:13:22.539837+0000","last_active":"2023-10-03T20:13:26.121641+0000","last_peered":"2023-10-03T20:13:26.121641+0000","last_clean":"2023-10-03T20:13:26.121641+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T20:13:26.121641+0000","last_undegraded":"2023-10-03T20:13:26.121641+0000","last_fullsized":"2023-10-03T20:13:26.121641+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:15:59.949043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T20:13:30.506 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-03T20:13:30.506 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-03T20:13:30.506 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-03T20:13:30.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-03T20:13:30.805 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T20:13:30.805 INFO:teuthology.orchestra.run.smithi167.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-03T20:13:30.822 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-03T20:13:30.822 INFO:teuthology.run_tasks:Running task exec... 2023-10-03T20:13:30.831 INFO:teuthology.task.exec:Executing custom commands... 2023-10-03T20:13:30.831 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T20:13:30.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2023-10-03T20:13:31.154 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'cache' created 2023-10-03T20:13:31.174 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2023-10-03T20:13:32.165 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2023-10-03T20:13:32.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2023-10-03T20:13:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:set cache-mode for pool 'cache' to writeback 2023-10-03T20:13:33.197 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2023-10-03T20:13:34.178 INFO:teuthology.orchestra.run.smithi167.stderr:overlay for 'rbd' is now (or already was) 'cache' 2023-10-03T20:13:34.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2023-10-03T20:13:35.184 INFO:teuthology.orchestra.run.smithi167.stderr:set pool 3 hit_set_type to bloom 2023-10-03T20:13:35.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2023-10-03T20:13:36.191 INFO:teuthology.orchestra.run.smithi167.stderr:set pool 3 hit_set_count to 8 2023-10-03T20:13:36.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2023-10-03T20:13:37.197 INFO:teuthology.orchestra.run.smithi167.stderr:set pool 3 hit_set_period to 60 2023-10-03T20:13:37.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2023-10-03T20:13:38.199 INFO:teuthology.orchestra.run.smithi167.stderr:set pool 3 target_max_objects to 250 2023-10-03T20:13:38.225 INFO:teuthology.run_tasks:Running task install... 2023-10-03T20:13:38.234 DEBUG:teuthology.task.install:project ceph 2023-10-03T20:13:38.234 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-03T20:13:38.235 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-03T20:13:38.235 INFO:teuthology.task.install:Using flavor: default 2023-10-03T20:13:38.240 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-03T20:13:38.240 INFO:teuthology.task.install:extra packages: [] 2023-10-03T20:13:38.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-key list | grep Ceph 2023-10-03T20:13:38.300 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-03T20:13:38.328 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-03T20:13:38.328 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph.com (release key) 2023-10-03T20:13:38.329 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-03T20:13:38.329 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-03T20:13:38.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T20:13:38.711 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default/ 2023-10-03T20:13:38.711 INFO:teuthology.task.install.deb:Package version is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:13:38.815 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:13:38.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-03T20:13:38.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-03T20:13:38.996 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T20:13:39.009 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-03T20:13:39.027 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-03T20:13:39.037 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-03T20:13:39.239 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy InRelease 2023-10-03T20:13:39.305 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release 2023-10-03T20:13:39.370 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-03T20:13:40.344 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:13:40.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6336-ga7de91bf-1jammy cephadm=18.0.0-6336-ga7de91bf-1jammy ceph-mds=18.0.0-6336-ga7de91bf-1jammy ceph-mgr=18.0.0-6336-ga7de91bf-1jammy ceph-common=18.0.0-6336-ga7de91bf-1jammy ceph-fuse=18.0.0-6336-ga7de91bf-1jammy ceph-test=18.0.0-6336-ga7de91bf-1jammy ceph-volume=18.0.0-6336-ga7de91bf-1jammy radosgw=18.0.0-6336-ga7de91bf-1jammy python3-rados=18.0.0-6336-ga7de91bf-1jammy python3-rgw=18.0.0-6336-ga7de91bf-1jammy python3-cephfs=18.0.0-6336-ga7de91bf-1jammy python3-rbd=18.0.0-6336-ga7de91bf-1jammy libcephfs2=18.0.0-6336-ga7de91bf-1jammy libcephfs-dev=18.0.0-6336-ga7de91bf-1jammy librados2=18.0.0-6336-ga7de91bf-1jammy librbd1=18.0.0-6336-ga7de91bf-1jammy rbd-fuse=18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:13:40.417 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:13:40.602 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-03T20:13:40.603 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-03T20:13:40.765 INFO:teuthology.orchestra.run.smithi167.stdout:ceph is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-common is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-fuse is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mds is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mgr is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-test is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-volume is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.766 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-dev is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs2 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:librados2 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:librbd1 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:python3-cephfs is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rados is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rbd is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rgw is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:radosgw is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T20:13:40.767 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T20:13:40.768 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T20:13:40.768 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-03T20:13:40.768 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-03T20:13:40.768 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T20:13:40.804 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 0 newly installed, 0 to remove and 254 not upgraded. 2023-10-03T20:13:40.804 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-03T20:13:40.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-03T20:13:40.864 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-03T20:13:41.052 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-03T20:13:41.052 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-03T20:13:41.211 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T20:13:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-03T20:13:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-03T20:13:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-03T20:13:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-03T20:13:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-03T20:13:41.212 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T20:13:41.213 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2023-10-03T20:13:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemblk1 2023-10-03T20:13:41.214 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2023-10-03T20:13:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: gnuplot gfio python-scipy 2023-10-03T20:13:41.239 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2023-10-03T20:13:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: fio libpmemblk1 2023-10-03T20:13:41.335 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 2 newly installed, 0 to remove and 254 not upgraded. 2023-10-03T20:13:41.335 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 4,112 kB of archives. 2023-10-03T20:13:41.335 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-03T20:13:41.335 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-03T20:13:41.482 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-03T20:13:42.373 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 4,112 kB in 1s (5,378 kB/s) 2023-10-03T20:13:42.574 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-03T20:13:42.678 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172896 files and directories currently installed.) 2023-10-03T20:13:42.684 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-03T20:13:42.764 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-03T20:13:43.101 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package fio. 2023-10-03T20:13:43.129 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-03T20:13:43.167 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking fio (3.28-1) ... 2023-10-03T20:13:43.703 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-03T20:13:43.831 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up fio (3.28-1) ... 2023-10-03T20:13:44.376 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-03T20:13:45.377 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-03T20:13:45.853 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-03T20:13:45.857 DEBUG:teuthology.parallel:result is None 2023-10-03T20:13:45.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T20:13:46.122 DEBUG:teuthology.orchestra.run.smithi167:> dpkg-query -W -f '${Version}' ceph 2023-10-03T20:13:46.142 INFO:teuthology.orchestra.run.smithi167.stdout:18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:13:46.142 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T20:13:46.142 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336-ga7de91bf-1jammy is installed. 2023-10-03T20:13:46.142 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-03T20:13:46.142 INFO:teuthology.run_tasks:Running task workunit... 2023-10-03T20:13:46.155 INFO:tasks.workunit:Pulling workunits from ref 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 2023-10-03T20:13:46.155 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-03T20:13:46.155 DEBUG:teuthology.orchestra.run.smithi167:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-03T20:13:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T20:13:46.191 INFO:teuthology.orchestra.run.smithi167.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-03T20:13:46.192 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-03T20:13:46.239 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-03T20:13:46.239 DEBUG:teuthology.orchestra.run.smithi167:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-03T20:13:46.287 INFO:tasks.workunit:timeout=3h 2023-10-03T20:13:46.287 INFO:tasks.workunit:cleanup=True 2023-10-03T20:13:46.287 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ajarr/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 2023-10-03T20:13:46.337 INFO:tasks.workunit.client.0.smithi167.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr:Note: switching to '7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231'. 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr:state without impacting any branches by switching back to a branch. 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-03T20:15:28.895 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr: git switch -c 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr:Or undo this operation with: 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr: git switch - 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr: 2023-10-03T20:15:28.896 INFO:tasks.workunit.client.0.smithi167.stderr:HEAD is now at 7fa2fe7fd78 rbd_support_module_reacovery.sh: check CLIENT_ADDR non-zero 2023-10-03T20:15:28.899 DEBUG:teuthology.orchestra.run.smithi167:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-03T20:15:28.947 INFO:tasks.workunit.client.0.smithi167.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-03T20:15:28.951 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T20:15:28.951 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-03T20:15:29.755 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-03T20:15:29.835 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-03T20:15:29.896 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T20:15:29.897 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T20:15:29.897 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-03T20:15:29.946 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T20:15:29.949 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-03T20:15:29.949 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-03T20:15:29.999 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-03T20:15:30.000 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-03T20:15:30.001 DEBUG:teuthology.orchestra.run.smithi167:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-03T20:15:30.053 INFO:tasks.workunit.client.0.smithi167.stderr:+ POOL=rbd 2023-10-03T20:15:30.054 INFO:tasks.workunit.client.0.smithi167.stderr:+ IMAGE_PREFIX=image 2023-10-03T20:15:30.054 INFO:tasks.workunit.client.0.smithi167.stderr:+ NUM_IMAGES=20 2023-10-03T20:15:30.054 INFO:tasks.workunit.client.0.smithi167.stderr:+ RUN_TIME=3600 2023-10-03T20:15:30.054 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror pool enable rbd image 2023-10-03T20:15:30.148 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-03T20:15:30.185 INFO:tasks.workunit.client.0.smithi167.stdout:0c72a868-8d3d-4a5a-bdbc-1d4f57668bc2 2023-10-03T20:15:30.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i = 1 )) 2023-10-03T20:15:30.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:30.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image1 2023-10-03T20:15:32.141 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-03T20:15:33.155 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:33.165 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-03T20:15:33.226 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:33.226 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:33.226 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image2 2023-10-03T20:15:33.279 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-03T20:15:34.149 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:34.160 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-03T20:15:34.220 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:34.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:34.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image3 2023-10-03T20:15:34.272 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-03T20:15:35.158 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:35.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-03T20:15:35.229 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:35.230 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:35.230 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image4 2023-10-03T20:15:35.277 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-03T20:15:36.177 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:36.187 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-03T20:15:36.248 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:36.248 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:36.248 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image5 2023-10-03T20:15:36.296 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-03T20:15:37.171 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:37.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-03T20:15:37.244 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:37.245 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:37.245 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image6 2023-10-03T20:15:37.292 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-03T20:15:38.188 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:38.198 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-03T20:15:38.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:38.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:38.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image7 2023-10-03T20:15:38.310 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-03T20:15:39.200 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:39.211 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-03T20:15:39.272 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:39.272 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:39.272 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image8 2023-10-03T20:15:39.324 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-03T20:15:40.131 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:40.141 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-03T20:15:40.200 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:40.200 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:40.201 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image9 2023-10-03T20:15:40.248 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-03T20:15:41.214 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:41.223 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-03T20:15:41.285 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:41.285 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:41.285 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image10 2023-10-03T20:15:41.333 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-03T20:15:42.214 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:42.224 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-03T20:15:42.283 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:42.283 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:42.283 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image11 2023-10-03T20:15:42.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-03T20:15:43.227 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:43.237 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-03T20:15:43.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:43.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:43.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image12 2023-10-03T20:15:43.345 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-03T20:15:44.240 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:44.248 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-03T20:15:44.309 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:44.309 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:44.309 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image13 2023-10-03T20:15:44.358 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-03T20:15:45.254 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:45.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-03T20:15:45.321 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:45.322 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:45.322 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image14 2023-10-03T20:15:45.371 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-03T20:15:46.262 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:46.271 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-03T20:15:46.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:46.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:46.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image15 2023-10-03T20:15:46.378 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-03T20:15:47.277 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:47.287 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-03T20:15:47.349 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:47.349 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:47.349 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image16 2023-10-03T20:15:47.396 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-03T20:15:48.284 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:48.293 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-03T20:15:48.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:48.358 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:48.358 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image17 2023-10-03T20:15:48.407 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-03T20:15:49.297 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:49.307 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-03T20:15:49.371 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:49.371 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:49.371 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image18 2023-10-03T20:15:49.420 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-03T20:15:50.293 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:50.299 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-03T20:15:50.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:50.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:50.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image19 2023-10-03T20:15:50.403 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-03T20:15:51.316 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:51.326 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-03T20:15:51.388 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:51.388 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:51.388 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd create --size 1G rbd/image20 2023-10-03T20:15:51.436 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-03T20:15:52.324 INFO:tasks.workunit.client.0.smithi167.stdout:Mirroring enabled 2023-10-03T20:15:52.332 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-03T20:15:52.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i = 1 )) 2023-10-03T20:15:52.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.393 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image1 2023-10-03T20:15:52.523 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-03T20:15:52.523 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.523 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.523 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:52.523 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image2 2023-10-03T20:15:52.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-03T20:15:52.594 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.594 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.594 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:52.594 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image3 2023-10-03T20:15:52.666 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-03T20:15:52.666 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.666 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.666 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:52.666 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image4 2023-10-03T20:15:52.738 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-03T20:15:52.738 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.739 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.739 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:52.739 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image5 2023-10-03T20:15:52.814 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-03T20:15:52.814 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.814 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.814 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:52.814 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image6 2023-10-03T20:15:52.928 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-03T20:15:52.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:52.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:52.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:52.929 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image7 2023-10-03T20:15:53.027 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-03T20:15:53.028 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.028 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.028 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.028 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image8 2023-10-03T20:15:53.155 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-03T20:15:53.155 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.155 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.155 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.155 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image9 2023-10-03T20:15:53.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-03T20:15:53.281 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.281 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.281 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.289 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image10 2023-10-03T20:15:53.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-03T20:15:53.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.421 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image11 2023-10-03T20:15:53.421 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-03T20:15:53.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.545 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image12 2023-10-03T20:15:53.687 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-03T20:15:53.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.688 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image13 2023-10-03T20:15:53.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-03T20:15:53.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:53.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:53.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:53.852 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image14 2023-10-03T20:15:54.046 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-03T20:15:54.047 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:54.047 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:54.047 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:54.047 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image15 2023-10-03T20:15:54.234 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-03T20:15:54.234 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:54.235 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:54.240 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:54.241 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image16 2023-10-03T20:15:54.417 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-03T20:15:54.417 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:54.417 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:54.417 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:54.417 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image17 2023-10-03T20:15:54.564 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-03T20:15:54.564 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:54.564 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:54.565 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image18 2023-10-03T20:15:54.574 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:54.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-03T20:15:54.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:54.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:54.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:54.714 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image19 2023-10-03T20:15:54.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-03T20:15:54.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:54.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:54.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:15:54.851 INFO:tasks.workunit.client.0.smithi167.stderr:++ sudo rbd device map rbd/image20 2023-10-03T20:15:55.068 INFO:tasks.workunit.client.0.smithi167.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-03T20:15:55.069 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T20:15:55.069 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 20 )) 2023-10-03T20:15:55.069 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364155 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ END_TIME=1696367755 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR= 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR= 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364155 -le 1696367755 ]] 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z '' ]] 2023-10-03T20:15:55.071 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:15:55.073 INFO:tasks.workunit.client.0.smithi167.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T20:16:00.772 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:00.768+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:16:00.772 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:00.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:16:00.772 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:00.768+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:03.827 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:03.824+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:03.829 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:03.824+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:16:03.830 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:03.828+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:16:05.080 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:16:05.080 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:16:05.081 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:16:05.084 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:16:06.285 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/265975161 2023-10-03T20:16:06.285 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:16:06.287 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364166 2023-10-03T20:16:06.287 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364166 -le 1696367755 ]] 2023-10-03T20:16:06.287 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/265975161 ]] 2023-10-03T20:16:06.288 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/265975161 != '' ]] 2023-10-03T20:16:06.288 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/265975161 2023-10-03T20:16:09.129 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/265975161 until 2023-10-03T21:16:08.443459+0000 (3600 sec) 2023-10-03T20:16:09.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:16:09.182 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/265975161 2023-10-03T20:16:09.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:09.996+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:10.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:10.252+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:16:10.270 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:10.256+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:16:10.270 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:10.256+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:16:10.270 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:10.256+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:16:10.270 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:10.256+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:16:10.270 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:10.256+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:16:10.532 INFO:tasks.workunit.client.0.smithi167.stderr:listed 1 entries 2023-10-03T20:16:10.604 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/265975161 2023-10-03T20:16:10.604 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:16:20.618 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:16:20.618 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:16:20.619 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:16:20.619 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:16:22.055 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/265975161 2023-10-03T20:16:22.055 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:16:22.058 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364182 2023-10-03T20:16:22.059 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364182 -le 1696367755 ]] 2023-10-03T20:16:22.059 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/265975161 ]] 2023-10-03T20:16:22.059 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/265975161 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\6\5\9\7\5\1\6\1 ]] 2023-10-03T20:16:22.059 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:16:24.526 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:24.520+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:24.527 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:24.520+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:16:26.393 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:26.388+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:16:26.393 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:26.388+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:16:31.306 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:31.304+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:16:31.342 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:31.336+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:16:31.342 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:31.336+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:16:31.342 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:31.336+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:16:31.344 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:31.340+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:31.344 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:31.340+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:16:31.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:31.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:16:32.061 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:16:32.062 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:16:32.062 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:16:32.084 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:16:33.062 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:33.048+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:16:33.062 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:33.048+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:16:33.063 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:33.048+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:16:33.578 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:16:33.580 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:16:33.581 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364193 2023-10-03T20:16:33.581 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364193 -le 1696367755 ]] 2023-10-03T20:16:33.581 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1903251917 ]] 2023-10-03T20:16:33.582 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1903251917 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\6\5\9\7\5\1\6\1 ]] 2023-10-03T20:16:33.582 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/1903251917 2023-10-03T20:16:34.084 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:34.076+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:16:34.088 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:34.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:16:34.088 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:34.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:16:34.096 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:34.088+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:16:35.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:35.832+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:35.838 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:35.832+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:16:36.193 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/1903251917 until 2023-10-03T21:16:35.594512+0000 (3600 sec) 2023-10-03T20:16:36.276 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/1903251917 2023-10-03T20:16:36.277 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:16:36.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.900+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:16:36.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.900+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:16:36.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.900+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:16:36.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.904+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:36.914 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.904+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:16:36.915 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.904+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:16:36.915 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.904+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:16:36.915 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.904+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:16:36.915 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:36.912+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:16:37.354 INFO:tasks.workunit.client.0.smithi167.stderr:listed 2 entries 2023-10-03T20:16:37.388 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:16:37.388 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:16:37.669 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:37.664+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:16:37.669 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:16:37.664+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:16:47.400 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:16:47.404 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:16:47.413 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:16:47.414 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:16:48.615 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:16:48.616 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:16:48.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364208 2023-10-03T20:16:48.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364208 -le 1696367755 ]] 2023-10-03T20:16:48.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1903251917 ]] 2023-10-03T20:16:48.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1903251917 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\0\3\2\5\1\9\1\7 ]] 2023-10-03T20:16:48.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:16:54.525 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:54.520+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:16:54.526 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:54.520+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:16:54.526 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:54.520+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:16:54.529 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:54.524+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:16:54.529 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:16:54.524+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:16:58.620 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:16:58.621 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:16:58.621 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:16:58.653 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:16:59.900 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:16:59.900 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:16:59.901 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364219 2023-10-03T20:16:59.901 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364219 -le 1696367755 ]] 2023-10-03T20:16:59.901 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1903251917 ]] 2023-10-03T20:16:59.902 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1903251917 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\0\3\2\5\1\9\1\7 ]] 2023-10-03T20:16:59.902 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:17:00.009 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.004+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.009 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.004+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156c100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156c300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156c900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.032+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.064 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.060+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.064 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.060+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.080+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.080+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.093 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.088+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.093 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.088+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.096+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.096+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e15ab300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.102 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.100+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.100+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e15ab500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.104+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.104+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.116 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.112+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.116 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.112+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.124 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.120+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.124 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.120+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.125 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.120+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.125 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.120+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.125 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.124+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:00.126 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:17:00.124+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e156d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:17:05.702 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:17:05.702 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:17:05.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:05.699+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:17:07.422 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:07.415+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:17:09.915 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:17:09.921 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:17:09.921 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:17:09.933 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:17:11.257 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:17:11.258 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:17:11.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364231 2023-10-03T20:17:11.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364231 -le 1696367755 ]] 2023-10-03T20:17:11.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1903251917 ]] 2023-10-03T20:17:11.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1903251917 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\0\3\2\5\1\9\1\7 ]] 2023-10-03T20:17:11.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:17:21.262 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:17:21.266 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:17:21.274 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:17:21.283 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:17:22.742 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:17:22.742 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:17:22.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364242 2023-10-03T20:17:22.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364242 -le 1696367755 ]] 2023-10-03T20:17:22.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1903251917 ]] 2023-10-03T20:17:22.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1903251917 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\0\3\2\5\1\9\1\7 ]] 2023-10-03T20:17:22.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:17:25.845 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:25.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:17:25.847 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:25.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:17:25.847 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:25.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:17:25.847 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:25.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:17:25.847 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:25.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:17:25.847 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:25.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:17:30.637 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.631+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:17:30.637 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.631+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:17:30.637 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.631+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:17:30.638 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.631+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:17:30.638 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.635+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:17:30.638 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.635+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:17:30.638 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.635+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:17:30.638 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:30.635+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:17:32.500 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:32.495+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:17:32.500 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:32.495+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:17:32.500 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:32.495+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:17:32.500 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:32.495+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:17:32.500 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:17:32.495+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:17:32.754 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:17:32.756 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:17:32.757 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:17:32.759 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:17:34.012 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1903251917 2023-10-03T20:17:34.012 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:17:34.012 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364254 2023-10-03T20:17:34.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364254 -le 1696367755 ]] 2023-10-03T20:17:34.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1903251917 ]] 2023-10-03T20:17:34.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1903251917 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\0\3\2\5\1\9\1\7 ]] 2023-10-03T20:17:34.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:17:44.015 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:17:44.015 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:17:44.036 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:17:44.048 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:17:45.515 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2476189247 2023-10-03T20:17:45.536 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:17:45.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364265 2023-10-03T20:17:45.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364265 -le 1696367755 ]] 2023-10-03T20:17:45.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2476189247 ]] 2023-10-03T20:17:45.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2476189247 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\0\3\2\5\1\9\1\7 ]] 2023-10-03T20:17:45.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2476189247 2023-10-03T20:17:45.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:45.771+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:17:45.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:45.771+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:17:45.777 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:45.775+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:17:45.778 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:45.775+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:17:48.187 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2476189247 until 2023-10-03T21:17:47.468301+0000 (3600 sec) 2023-10-03T20:17:48.214 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2476189247 2023-10-03T20:17:48.214 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:17:49.545 INFO:tasks.workunit.client.0.smithi167.stderr:listed 3 entries 2023-10-03T20:17:49.591 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2476189247 2023-10-03T20:17:49.591 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:17:56.559 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:56.555+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:17:56.560 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:17:56.555+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:17:59.597 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:17:59.599 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:17:59.606 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:17:59.610 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:18:00.009 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.003+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.009 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.003+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.031 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.027+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.031 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.027+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.031 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.027+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.031 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.027+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1863b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.039+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.039+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1863d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.044 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.043+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.044 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.043+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.087+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.087+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.087+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.087+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1980100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.095 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.087+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1980300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.095 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.087+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1980500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.097 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.095+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.097 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.095+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.097 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.095+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e0b44680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.097 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.095+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e15aaa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.099+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.099+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e159ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1265000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.107 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.107 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.107 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.107 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1980100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.107 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.107 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.103+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1863e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.107+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.107+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1980580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.143 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.139+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.144 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.139+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e0b44680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.144 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.139+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.144 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:00.139+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e15aaa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:00.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:00.187+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:18:01.210 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2476189247 2023-10-03T20:18:01.219 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:18:01.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364281 2023-10-03T20:18:01.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364281 -le 1696367755 ]] 2023-10-03T20:18:01.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2476189247 ]] 2023-10-03T20:18:01.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2476189247 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\7\6\1\8\9\2\4\7 ]] 2023-10-03T20:18:01.221 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:18:11.226 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:18:11.227 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:18:11.230 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:18:11.231 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:18:12.673 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2476189247 2023-10-03T20:18:12.673 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:18:12.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364292 2023-10-03T20:18:12.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364292 -le 1696367755 ]] 2023-10-03T20:18:12.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2476189247 ]] 2023-10-03T20:18:12.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2476189247 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\7\6\1\8\9\2\4\7 ]] 2023-10-03T20:18:12.675 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:18:21.282 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:21.278+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:18:21.290 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:21.282+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:18:22.681 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:18:22.683 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:18:22.685 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:18:22.707 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:18:23.875 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:23.870+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:18:23.885 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:23.870+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:18:24.039 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2476189247 2023-10-03T20:18:24.046 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:18:24.052 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364304 2023-10-03T20:18:24.052 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364304 -le 1696367755 ]] 2023-10-03T20:18:24.052 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2476189247 ]] 2023-10-03T20:18:24.052 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2476189247 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\7\6\1\8\9\2\4\7 ]] 2023-10-03T20:18:24.052 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:18:25.471 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.462+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:18:25.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.466+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:18:25.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.466+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:18:25.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.466+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:18:25.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.466+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:18:25.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.466+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:18:25.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:25.466+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:18:34.057 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:18:34.068 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:18:34.084 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:18:34.090 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:18:35.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2476189247 2023-10-03T20:18:35.619 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:18:35.620 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364315 2023-10-03T20:18:35.620 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364315 -le 1696367755 ]] 2023-10-03T20:18:35.620 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2476189247 ]] 2023-10-03T20:18:35.620 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2476189247 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\7\6\1\8\9\2\4\7 ]] 2023-10-03T20:18:35.620 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:18:40.273 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.266+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:18:40.277 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:18:40.277 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:18:40.277 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:18:40.277 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:18:40.277 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:18:40.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:18:40.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:18:40.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:18:40.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:18:40.280 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:40.274+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:18:41.081 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:18:41.074+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:18:45.626 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:18:45.633 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:18:45.636 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:18:45.637 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:18:46.998 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2544279431 2023-10-03T20:18:47.005 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:18:47.012 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364327 2023-10-03T20:18:47.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364327 -le 1696367755 ]] 2023-10-03T20:18:47.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2544279431 ]] 2023-10-03T20:18:47.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2544279431 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\7\6\1\8\9\2\4\7 ]] 2023-10-03T20:18:47.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2544279431 2023-10-03T20:18:47.409 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.406+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:18:47.413 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.410+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:18:47.413 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.410+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:18:47.413 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.410+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:18:47.413 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.410+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:18:47.414 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.410+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:18:47.414 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:47.410+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:18:48.261 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:48.254+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:48.261 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:18:48.254+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e15aaa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:18:49.247 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2544279431 until 2023-10-03T21:18:48.604779+0000 (3600 sec) 2023-10-03T20:18:49.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2544279431 2023-10-03T20:18:49.313 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:18:50.493 INFO:tasks.workunit.client.0.smithi167.stderr:listed 4 entries 2023-10-03T20:18:50.531 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2544279431 2023-10-03T20:18:50.532 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:18:58.020 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:58.018+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:18:58.020 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:58.018+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:18:58.020 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:58.018+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:18:58.021 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:18:58.018+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:19:00.143 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:00.142+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:19:00.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:00.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55692a1dd040 2023-10-03T20:19:00.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:00.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55693115c680 2023-10-03T20:19:00.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:00.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55693115cc30 2023-10-03T20:19:00.554 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:19:00.555 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:19:00.555 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:19:00.555 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:19:00.694 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:00.686+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:19:00.695 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:00.686+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:19:01.721 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:19:01.714+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:19:01.721 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:19:01.718+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:19:01.722 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:19:01.718+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:19:01.722 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:19:01.718+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:19:01.722 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:19:01.718+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:19:01.738 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:19:01.755 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:19:01.757 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364341 2023-10-03T20:19:01.757 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364341 -le 1696367755 ]] 2023-10-03T20:19:01.758 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/197978149 ]] 2023-10-03T20:19:01.758 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/197978149 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\5\4\4\2\7\9\4\3\1 ]] 2023-10-03T20:19:01.758 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/197978149 2023-10-03T20:19:03.194 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::ImageWatcher: 0x5606e136bb00 image watch failed: 94587558787072, (107) Transport endpoint is not connected 2023-10-03T20:19:03.194 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::Watcher: 0x5606e136bb00 handle_error: handle=94587558787072: (107) Transport endpoint is not connected 2023-10-03T20:19:03.195 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::ImageWatcher: 0x5606e0c05b00 image watch failed: 94587558788096, (107) Transport endpoint is not connected 2023-10-03T20:19:03.195 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::Watcher: 0x5606e0c05b00 handle_error: handle=94587558788096: (107) Transport endpoint is not connected 2023-10-03T20:19:03.195 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::watcher::RewatchRequest: 0x5606e0de0230 handle_unwatch client blocklisted 2023-10-03T20:19:03.195 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::Watcher: 0x5606e136bb00 handle_rewatch: client blocklisted 2023-10-03T20:19:03.195 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::watcher::RewatchRequest: 0x5606e0de0e10 handle_unwatch client blocklisted 2023-10-03T20:19:03.196 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::Watcher: 0x5606e0c05b00 handle_rewatch: client blocklisted 2023-10-03T20:19:03.196 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::ImageWatcher: 0x5606e136b800 image watch failed: 94587558784000, (107) Transport endpoint is not connected 2023-10-03T20:19:03.196 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.194+0000 7f6ecc29e640 -1 librbd::Watcher: 0x5606e136b800 handle_error: handle=94587558784000: (107) Transport endpoint is not connected 2023-10-03T20:19:03.197 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.198+0000 7f6ecc29e640 -1 librbd::watcher::RewatchRequest: 0x5606e15e3860 handle_unwatch client blocklisted 2023-10-03T20:19:03.197 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:03.198+0000 7f6ecc29e640 -1 librbd::Watcher: 0x5606e136b800 handle_rewatch: client blocklisted 2023-10-03T20:19:04.187 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/197978149 until 2023-10-03T21:19:03.395844+0000 (3600 sec) 2023-10-03T20:19:04.196 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.194+0000 7f6ecca9f640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.196 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.194+0000 7f6ecca9f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5606e122d520 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.196 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.194+0000 7f6ecca9f640 -1 librbd::io::AioCompletion: 0x5606e18d0000 fail: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.198+0000 7f6ecca9f640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.198+0000 7f6ecca9f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5606e122c820 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.198+0000 7f6ecca9f640 -1 librbd::io::AioCompletion: 0x5606e1c8b600 fail: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.198+0000 7f6ecca9f640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.198+0000 7f6ecca9f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5606e122d450 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:19:04.198+0000 7f6ecca9f640 -1 librbd::io::AioCompletion: 0x5606e1cb5760 fail: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:19:04.227 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:19:04.230 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/197978149 2023-10-03T20:19:05.229 INFO:tasks.workunit.client.0.smithi167.stderr:listed 5 entries 2023-10-03T20:19:05.258 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:19:05.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:19:15.264 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:19:15.290 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:19:15.304 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:19:15.309 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:19:16.616 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:19:16.616 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:19:16.617 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364356 2023-10-03T20:19:16.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364356 -le 1696367755 ]] 2023-10-03T20:19:16.622 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/197978149 ]] 2023-10-03T20:19:16.622 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/197978149 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\7\9\7\8\1\4\9 ]] 2023-10-03T20:19:16.623 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:19:26.620 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:19:26.621 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:19:26.621 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:19:26.626 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:19:27.776 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:19:27.776 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:19:27.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364367 2023-10-03T20:19:27.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364367 -le 1696367755 ]] 2023-10-03T20:19:27.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/197978149 ]] 2023-10-03T20:19:27.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/197978149 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\7\9\7\8\1\4\9 ]] 2023-10-03T20:19:27.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:19:37.785 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:19:37.787 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:19:37.790 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:19:37.792 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:19:39.343 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:19:39.343 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:19:39.344 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364379 2023-10-03T20:19:39.344 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364379 -le 1696367755 ]] 2023-10-03T20:19:39.344 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/197978149 ]] 2023-10-03T20:19:39.344 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/197978149 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\7\9\7\8\1\4\9 ]] 2023-10-03T20:19:39.344 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:19:45.370 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:45.369+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:19:45.370 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:45.369+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:19:46.878 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:19:46.873+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:19:49.357 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:19:49.357 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:19:49.361 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:19:49.379 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:19:49.833 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:49.829+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:19:49.833 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:19:49.829+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:19:50.693 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:19:50.704 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:19:50.705 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364390 2023-10-03T20:19:50.706 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364390 -le 1696367755 ]] 2023-10-03T20:19:50.706 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/197978149 ]] 2023-10-03T20:19:50.706 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/197978149 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\7\9\7\8\1\4\9 ]] 2023-10-03T20:19:50.706 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:20:00.715 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:20:00.715 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:20:00.716 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:20:00.730 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:20:01.045 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:20:01.045 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:20:01.045 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:20:01.045 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:20:01.046 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:20:01.046 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:20:01.046 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:01.045+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:20:02.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/197978149 2023-10-03T20:20:02.109 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:20:02.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364402 2023-10-03T20:20:02.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364402 -le 1696367755 ]] 2023-10-03T20:20:02.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/197978149 ]] 2023-10-03T20:20:02.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/197978149 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\7\9\7\8\1\4\9 ]] 2023-10-03T20:20:02.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:20:03.794 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:03.793+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:20:03.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:03.813+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:20:12.115 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:20:12.115 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:20:12.117 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:20:12.126 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:20:13.481 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:20:13.482 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:20:13.483 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364413 2023-10-03T20:20:13.483 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364413 -le 1696367755 ]] 2023-10-03T20:20:13.483 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3592042819 ]] 2023-10-03T20:20:13.483 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3592042819 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\9\7\9\7\8\1\4\9 ]] 2023-10-03T20:20:13.483 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/3592042819 2023-10-03T20:20:16.294 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/3592042819 until 2023-10-03T21:20:15.694376+0000 (3600 sec) 2023-10-03T20:20:16.343 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/3592042819 2023-10-03T20:20:16.350 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:20:17.711 INFO:tasks.workunit.client.0.smithi167.stderr:listed 6 entries 2023-10-03T20:20:17.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:20:17.755 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:20:27.758 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:20:27.765 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:20:27.779 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:20:27.781 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:20:28.978 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:20:28.979 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:20:28.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364428 2023-10-03T20:20:28.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364428 -le 1696367755 ]] 2023-10-03T20:20:28.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3592042819 ]] 2023-10-03T20:20:28.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3592042819 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\5\9\2\0\4\2\8\1\9 ]] 2023-10-03T20:20:28.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:20:38.685 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:38.684+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:20:38.980 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:20:38.981 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:20:38.987 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:20:38.993 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:20:40.188 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:20:40.188 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:20:40.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364440 2023-10-03T20:20:40.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364440 -le 1696367755 ]] 2023-10-03T20:20:40.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3592042819 ]] 2023-10-03T20:20:40.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3592042819 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\5\9\2\0\4\2\8\1\9 ]] 2023-10-03T20:20:40.189 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:20:43.830 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:43.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:20:43.830 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:43.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:20:44.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:20:45.138 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:20:45.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:20:45.140 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:44.368+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:20:50.194 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:20:50.196 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:20:50.203 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:20:50.210 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:20:51.713 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:20:51.718 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:20:51.728 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364451 2023-10-03T20:20:51.728 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364451 -le 1696367755 ]] 2023-10-03T20:20:51.728 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3592042819 ]] 2023-10-03T20:20:51.728 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3592042819 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\5\9\2\0\4\2\8\1\9 ]] 2023-10-03T20:20:51.729 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:20:55.241 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:55.236+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:20:55.241 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:55.236+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:20:55.242 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:55.236+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:20:55.242 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:55.240+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:20:55.242 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:55.240+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:20:55.242 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:20:55.240+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:20:57.161 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:57.160+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:20:57.166 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:57.164+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:20:57.166 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:20:57.164+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:21:00.015 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.012+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.012+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d40700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d40900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.032+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.032+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d40b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.039 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.036+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.039 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.036+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d40d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.040+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.040+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d40f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.044+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.044+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d41b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.044+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.052 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.044+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.052 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.044+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d41d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.052 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.044+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d41f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.062 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.060+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.062 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.060+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfc100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.092 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.088+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.092 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.088+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.097 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.097 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.098 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfc500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfc900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfcb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfcd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfcf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.100 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.092+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfd100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.127 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.124+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.128 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.124+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.134 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.132+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.134 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.132+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.135 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.132+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfd500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:00.135 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:21:00.132+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfde80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:21:01.600 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:01.596+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:21:01.600 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:01.596+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:21:01.600 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:01.596+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:21:01.600 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:01.600+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:21:01.624 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:01.620+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:21:01.741 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:21:01.752 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:21:01.763 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:21:01.775 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:21:03.239 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:21:03.240 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:21:03.242 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364463 2023-10-03T20:21:03.242 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364463 -le 1696367755 ]] 2023-10-03T20:21:03.242 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3592042819 ]] 2023-10-03T20:21:03.242 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3592042819 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\5\9\2\0\4\2\8\1\9 ]] 2023-10-03T20:21:03.242 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:21:03.324 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:03.312+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:21:03.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:03.312+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:21:03.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:03.312+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:21:03.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:03.312+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:21:03.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:03.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:21:03.326 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:03.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:21:13.243 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:21:13.243 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:21:13.247 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:21:13.258 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:21:14.473 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3592042819 2023-10-03T20:21:14.473 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:21:14.474 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364474 2023-10-03T20:21:14.474 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364474 -le 1696367755 ]] 2023-10-03T20:21:14.474 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3592042819 ]] 2023-10-03T20:21:14.474 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3592042819 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\5\9\2\0\4\2\8\1\9 ]] 2023-10-03T20:21:14.475 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:21:14.811 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:21:14.812 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:21:14.812 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:21:14.812 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:21:14.812 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:21:14.813 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:21:14.813 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:21:14.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:21:14.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:21:14.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:21:14.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:21:14.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:21:14.815 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:14.808+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:21:19.968 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:19.964+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:21:19.968 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:19.964+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:21:20.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:20.560+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:21:20.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:20.564+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:21:24.485 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:21:24.502 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:21:24.506 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:21:24.507 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:21:26.028 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:21:26.048 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:21:26.049 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364486 2023-10-03T20:21:26.049 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364486 -le 1696367755 ]] 2023-10-03T20:21:26.049 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/593542325 ]] 2023-10-03T20:21:26.049 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/593542325 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\5\9\2\0\4\2\8\1\9 ]] 2023-10-03T20:21:26.049 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/593542325 2023-10-03T20:21:26.223 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:26.219+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:21:26.265 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:26.263+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:21:26.269 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:26.267+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:21:26.387 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:26.383+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:21:26.388 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:26.383+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:21:26.388 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:26.387+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:21:28.266 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/593542325 until 2023-10-03T21:21:27.563749+0000 (3600 sec) 2023-10-03T20:21:28.302 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/593542325 2023-10-03T20:21:28.306 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:21:29.546 INFO:tasks.workunit.client.0.smithi167.stderr:listed 7 entries 2023-10-03T20:21:29.579 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:21:29.579 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:21:34.420 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.415+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:21:34.420 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.415+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:21:34.420 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.415+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:21:34.421 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.415+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:21:34.421 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.419+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:21:34.421 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.419+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:21:34.421 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:34.419+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:21:36.341 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:36.339+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:21:37.586 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:37.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:21:37.587 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:37.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:21:39.585 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:21:39.585 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:21:39.586 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:21:39.595 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:21:41.165 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:21:41.166 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:21:41.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364501 2023-10-03T20:21:41.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364501 -le 1696367755 ]] 2023-10-03T20:21:41.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/593542325 ]] 2023-10-03T20:21:41.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/593542325 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\9\3\5\4\2\3\2\5 ]] 2023-10-03T20:21:41.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:21:43.803 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:43.799+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:21:43.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:43.803+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:21:44.111 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:44.099+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:21:44.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:44.827+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:21:46.243 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.239+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:21:46.244 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.239+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:21:46.244 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.239+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:21:46.249 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.243+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:21:46.249 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.247+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:21:46.249 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.247+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:21:46.365 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.355+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:21:46.365 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.355+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:21:46.365 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.355+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:21:46.365 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:21:46.355+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:21:51.170 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:21:51.170 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:21:51.175 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:21:51.188 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:21:52.178 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:52.171+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:21:52.189 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:52.183+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:21:52.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:52.215+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:21:52.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:52.215+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:21:52.222 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:52.219+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:21:52.222 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:21:52.219+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:21:52.599 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:21:52.599 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:21:52.599 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364512 2023-10-03T20:21:52.599 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364512 -le 1696367755 ]] 2023-10-03T20:21:52.600 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/593542325 ]] 2023-10-03T20:21:52.600 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/593542325 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\9\3\5\4\2\3\2\5 ]] 2023-10-03T20:21:52.600 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:22:00.006 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.003+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.006 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.003+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.007 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.003+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfd800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.007 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.003+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfda00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.015 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.011+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.015 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.011+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfdc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.023 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.019+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.023 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.019+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.031+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.031+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1c100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.039+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.039+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1c300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.039+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.039+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.043+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.043+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1c900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.112 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.107+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.114 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.111+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.114 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.111+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.199 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1d700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:00.201 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:22:00.195+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d1df00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:22:02.600 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:22:02.601 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:22:02.601 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:22:02.603 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:22:04.240 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:22:04.248 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:22:04.249 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364524 2023-10-03T20:22:04.249 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364524 -le 1696367755 ]] 2023-10-03T20:22:04.249 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/593542325 ]] 2023-10-03T20:22:04.249 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/593542325 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\9\3\5\4\2\3\2\5 ]] 2023-10-03T20:22:04.250 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:22:04.775 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:04.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:22:04.775 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:04.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:22:04.775 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:04.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:22:04.782 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:04.775+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:22:08.880 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:08.879+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:22:08.888 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:08.879+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:22:08.888 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:08.879+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:22:10.645 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.643+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:22:10.645 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.643+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:22:10.645 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.643+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:22:10.667 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:22:10.667 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:22:10.668 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:10.655+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:22:13.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:13.463+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:22:13.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:13.463+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:22:13.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:13.463+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:22:13.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:13.463+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:22:13.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:13.463+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:22:13.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:13.463+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:22:14.257 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:22:14.257 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:22:14.272 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:22:14.274 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:22:15.557 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:22:15.559 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:22:15.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364535 2023-10-03T20:22:15.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364535 -le 1696367755 ]] 2023-10-03T20:22:15.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/593542325 ]] 2023-10-03T20:22:15.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/593542325 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\9\3\5\4\2\3\2\5 ]] 2023-10-03T20:22:15.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:22:22.970 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:22.967+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:22:22.971 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:22.967+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:22:22.971 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:22.967+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:22:22.971 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:22.967+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:22:22.971 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:22.967+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:22:23.581 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:23.575+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:22:23.584 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:23.579+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:22:23.590 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:23.587+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:22:23.591 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:23.587+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:22:23.591 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:23.587+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:22:25.573 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:22:25.578 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:22:25.578 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:22:25.606 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:22:26.930 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:26.927+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:22:26.935 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:26.931+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:22:27.031 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/593542325 2023-10-03T20:22:27.037 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:22:27.043 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364547 2023-10-03T20:22:27.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364547 -le 1696367755 ]] 2023-10-03T20:22:27.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/593542325 ]] 2023-10-03T20:22:27.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/593542325 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\9\3\5\4\2\3\2\5 ]] 2023-10-03T20:22:27.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:22:30.160 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:30.159+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:22:30.172 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:30.163+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:22:30.173 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:30.163+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:22:31.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.099+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:22:31.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.099+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:22:31.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.099+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:22:31.106 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.099+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:22:31.106 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.099+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:22:31.112 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.107+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:22:31.112 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:22:31.112 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:22:31.112 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:22:31.112 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:31.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:22:37.055 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:22:37.056 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:22:37.060 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:22:37.074 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:22:37.485 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:22:37.483+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:22:38.679 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:22:38.686 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:22:38.687 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364558 2023-10-03T20:22:38.687 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364558 -le 1696367755 ]] 2023-10-03T20:22:38.687 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2415343388 ]] 2023-10-03T20:22:38.687 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2415343388 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\9\3\5\4\2\3\2\5 ]] 2023-10-03T20:22:38.687 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2415343388 2023-10-03T20:22:38.849 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:38.839+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:22:38.849 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:38.839+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:22:38.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:38.839+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:22:38.860 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:38.847+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:22:38.860 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:38.847+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:22:38.860 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:38.847+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:22:39.200 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:39.195+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:22:41.317 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2415343388 until 2023-10-03T21:22:40.581149+0000 (3600 sec) 2023-10-03T20:22:41.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:22:41.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2415343388 2023-10-03T20:22:42.567 INFO:tasks.workunit.client.0.smithi167.stderr:listed 8 entries 2023-10-03T20:22:42.623 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:22:42.623 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:22:52.626 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:22:52.626 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:22:52.641 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:22:52.641 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:22:53.830 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:22:53.834 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:22:53.836 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364573 2023-10-03T20:22:53.836 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364573 -le 1696367755 ]] 2023-10-03T20:22:53.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2415343388 ]] 2023-10-03T20:22:53.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2415343388 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\1\5\3\4\3\3\8\8 ]] 2023-10-03T20:22:53.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:22:57.661 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:57.658+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:22:57.663 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:57.658+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:22:57.663 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:57.658+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:22:57.663 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:22:57.658+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:23:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.010+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.010+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.010+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.010+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1863080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.010+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e159b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.010+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfdf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.020 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.018+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.020 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.018+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e0b45200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1bfd780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c62e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.043 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.043 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.038+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c62f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.076 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.074+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.077 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.074+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.082+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.082+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.090 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.086+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.090 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.086+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c62c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.090+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.090+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.094+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.094+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.101 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.098+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.101 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.098+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.102+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.102+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c63e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.109 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.106+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.109 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.106+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c90080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.113 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.110+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.113 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.110+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c90280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.125 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.122+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.125 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.122+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c90480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.140 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.134+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.140 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.134+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.140 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.134+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c90e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:00.140 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:23:00.134+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1c90d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:23:03.849 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:23:03.850 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:23:03.861 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:23:03.862 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:23:05.405 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:23:05.418 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:23:05.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364585 2023-10-03T20:23:05.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364585 -le 1696367755 ]] 2023-10-03T20:23:05.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2415343388 ]] 2023-10-03T20:23:05.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2415343388 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\1\5\3\4\3\3\8\8 ]] 2023-10-03T20:23:05.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:23:05.820 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:05.810+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:23:05.820 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:05.810+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:23:12.660 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:12.658+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:12.665 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:12.662+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:15.423 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:23:15.424 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:23:15.424 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:23:15.424 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:23:16.119 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:16.114+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:23:16.124 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:16.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:23:16.124 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:16.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:23:16.124 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:16.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:16.124 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:16.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:23:16.124 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:16.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:16.893 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:23:16.908 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:23:16.909 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364596 2023-10-03T20:23:16.909 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364596 -le 1696367755 ]] 2023-10-03T20:23:16.909 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2415343388 ]] 2023-10-03T20:23:16.909 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2415343388 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\1\5\3\4\3\3\8\8 ]] 2023-10-03T20:23:16.909 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:23:19.720 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:19.714+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:19.723 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:19.718+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:23:19.723 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:19.718+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:26.916 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:23:26.918 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:23:26.918 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:23:26.919 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:23:28.332 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:23:28.332 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:23:28.335 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364608 2023-10-03T20:23:28.335 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364608 -le 1696367755 ]] 2023-10-03T20:23:28.336 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2415343388 ]] 2023-10-03T20:23:28.336 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2415343388 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\1\5\3\4\3\3\8\8 ]] 2023-10-03T20:23:28.336 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:23:32.150 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:23:32.150 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:23:32.150 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:23:32.150 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:23:32.150 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:23:32.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:32.146+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:23:32.928 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.926+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:23:32.931 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.926+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:23:32.932 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.930+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:23:32.932 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.930+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:23:32.932 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.930+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:32.932 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.930+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:23:32.932 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:32.930+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:38.338 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:23:38.338 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:23:38.340 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:23:38.356 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:23:39.825 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2415343388 2023-10-03T20:23:39.826 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:23:39.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364619 2023-10-03T20:23:39.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364619 -le 1696367755 ]] 2023-10-03T20:23:39.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2415343388 ]] 2023-10-03T20:23:39.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2415343388 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\1\5\3\4\3\3\8\8 ]] 2023-10-03T20:23:39.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:23:40.433 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.426+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:23:40.435 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:23:40.435 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:23:40.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:23:40.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:40.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:23:40.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:40.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:43.517 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:43.514+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:43.517 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:43.514+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:45.640 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:45.626+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:23:45.652 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:45.650+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:23:46.465 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:46.462+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:23:46.465 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:46.462+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:23:46.466 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:46.462+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:23:46.467 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:46.462+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:23:46.468 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:23:46.466+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:23:48.483 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:23:48.483 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:23:48.483 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:23:48.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:23:48.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:23:48.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:23:48.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:48.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:23:49.832 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:23:49.836 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:23:49.836 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:23:49.847 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:23:51.223 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:23:51.224 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:23:51.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364631 2023-10-03T20:23:51.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364631 -le 1696367755 ]] 2023-10-03T20:23:51.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3020862503 ]] 2023-10-03T20:23:51.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3020862503 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\1\5\3\4\3\3\8\8 ]] 2023-10-03T20:23:51.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/3020862503 2023-10-03T20:23:53.389 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/3020862503 until 2023-10-03T21:23:52.702190+0000 (3600 sec) 2023-10-03T20:23:53.460 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:23:53.473 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/3020862503 2023-10-03T20:23:54.729 INFO:tasks.workunit.client.0.smithi167.stderr:listed 9 entries 2023-10-03T20:23:54.766 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:23:54.767 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:23:56.400 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:56.394+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:23:56.400 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:56.394+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:23:56.401 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:56.394+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:23:56.401 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:23:56.398+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1a80e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c63e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c62b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c63d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.050 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.050+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.055 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.054+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.055 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.054+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.083 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.083 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.083 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.084 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1a80e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c63e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c62b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c63d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.082+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.095 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.094+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.095 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.094+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.115 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.114+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:00.115 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:24:00.114+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:24:04.776 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:24:04.777 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:24:04.786 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:24:04.787 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:24:06.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:24:06.393 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:24:06.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364646 2023-10-03T20:24:06.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364646 -le 1696367755 ]] 2023-10-03T20:24:06.394 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3020862503 ]] 2023-10-03T20:24:06.394 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3020862503 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\0\2\0\8\6\2\5\0\3 ]] 2023-10-03T20:24:06.394 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:24:09.235 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:09.233+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:24:15.177 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:15.173+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:24:15.177 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:15.173+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:24:15.177 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:15.173+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:24:15.177 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:15.173+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:24:15.177 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:15.173+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:15.177 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:15.173+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:24:16.399 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:24:16.409 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:24:16.417 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:24:16.433 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:24:17.825 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:24:17.825 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:24:17.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364657 2023-10-03T20:24:17.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364657 -le 1696367755 ]] 2023-10-03T20:24:17.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3020862503 ]] 2023-10-03T20:24:17.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3020862503 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\0\2\0\8\6\2\5\0\3 ]] 2023-10-03T20:24:17.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:24:22.770 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.765+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:24:22.770 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.765+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:24:22.771 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.769+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:24:22.780 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.781+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:24:22.791 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.785+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:24:22.791 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.785+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:24:22.791 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:22.785+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:24:25.310 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:25.305+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:24:25.311 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:25.305+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:24:25.311 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:25.305+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:24:25.312 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:25.313+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:24:25.312 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:25.313+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:25.312 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:25.313+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:24:26.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:26.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:24:26.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:26.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:24:26.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:26.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:24:26.334 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:26.333+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:24:26.335 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:26.333+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:26.338 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:26.337+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:24:27.823 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:24:27.823 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:24:27.823 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:24:27.829 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:24:28.457 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:24:28.457 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:24:28.457 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:24:28.458 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:24:28.458 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:24:28.459 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:28.459 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:28.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:24:29.075 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:24:29.075 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:24:29.076 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364669 2023-10-03T20:24:29.077 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364669 -le 1696367755 ]] 2023-10-03T20:24:29.077 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3020862503 ]] 2023-10-03T20:24:29.077 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3020862503 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\0\2\0\8\6\2\5\0\3 ]] 2023-10-03T20:24:29.077 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:24:39.110 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:24:39.135 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:24:39.142 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:24:39.154 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:24:39.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:39.877+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:24:39.893 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:24:39.893+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:24:40.638 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:24:40.638 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:24:40.639 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364680 2023-10-03T20:24:40.639 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364680 -le 1696367755 ]] 2023-10-03T20:24:40.639 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3020862503 ]] 2023-10-03T20:24:40.639 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3020862503 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\0\2\0\8\6\2\5\0\3 ]] 2023-10-03T20:24:40.639 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:24:41.320 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:41.313+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:24:41.321 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:41.317+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:24:41.327 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:41.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:24:41.327 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:41.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:41.327 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:41.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:24:44.678 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:44.665+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:24:44.678 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:44.665+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:24:44.680 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:44.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:24:44.680 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:44.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:24:44.680 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:24:44.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:24:50.651 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:24:50.655 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:24:50.655 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:24:50.664 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:24:52.043 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/3020862503 2023-10-03T20:24:52.043 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:24:52.043 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364692 2023-10-03T20:24:52.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364692 -le 1696367755 ]] 2023-10-03T20:24:52.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/3020862503 ]] 2023-10-03T20:24:52.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/3020862503 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\0\2\0\8\6\2\5\0\3 ]] 2023-10-03T20:24:52.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:25:02.046 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:25:02.046 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:25:02.046 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:25:02.048 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:25:03.353 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2855518517 2023-10-03T20:25:03.353 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:25:03.353 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364703 2023-10-03T20:25:03.354 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364703 -le 1696367755 ]] 2023-10-03T20:25:03.354 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2855518517 ]] 2023-10-03T20:25:03.354 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2855518517 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\0\2\0\8\6\2\5\0\3 ]] 2023-10-03T20:25:03.354 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2855518517 2023-10-03T20:25:04.454 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:25:04.453+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:25:04.454 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:25:04.453+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1d1dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:25:05.449 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2855518517 until 2023-10-03T21:25:04.849806+0000 (3600 sec) 2023-10-03T20:25:05.528 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2855518517 2023-10-03T20:25:05.530 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:25:06.713 INFO:tasks.workunit.client.0.smithi167.stderr:listed 10 entries 2023-10-03T20:25:06.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2855518517 2023-10-03T20:25:06.755 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:25:16.757 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:25:16.758 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:25:16.762 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:25:16.770 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:25:18.203 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:25:18.204 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:25:18.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364718 2023-10-03T20:25:18.205 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364718 -le 1696367755 ]] 2023-10-03T20:25:18.205 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/4212689945 ]] 2023-10-03T20:25:18.205 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/4212689945 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\8\5\5\5\1\8\5\1\7 ]] 2023-10-03T20:25:18.205 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/4212689945 2023-10-03T20:25:20.330 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/4212689945 until 2023-10-03T21:25:19.625517+0000 (3600 sec) 2023-10-03T20:25:20.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:25:20.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/4212689945 2023-10-03T20:25:21.501 INFO:tasks.workunit.client.0.smithi167.stderr:listed 11 entries 2023-10-03T20:25:21.552 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:25:21.553 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:25:22.961 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:22.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:25:22.961 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:22.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:25:22.962 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:22.961+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:25:22.962 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:22.961+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:25:22.962 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:22.961+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:25:30.438 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:30.437+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:25:30.446 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:30.445+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:25:31.581 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:25:31.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:25:31.602 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:25:31.602 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:25:32.935 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:25:32.935 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:25:32.935 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364732 2023-10-03T20:25:32.936 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364732 -le 1696367755 ]] 2023-10-03T20:25:32.936 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/4212689945 ]] 2023-10-03T20:25:32.936 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/4212689945 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\4\2\1\2\6\8\9\9\4\5 ]] 2023-10-03T20:25:32.936 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:25:36.960 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:36.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:25:36.970 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:36.969+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:25:36.971 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:36.969+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:25:37.013 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:37.013+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:25:42.942 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:25:42.943 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:25:42.943 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:25:42.944 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:25:43.832 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:43.828+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:25:43.833 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:43.828+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:25:44.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:25:44.237 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:25:44.237 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364744 2023-10-03T20:25:44.237 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364744 -le 1696367755 ]] 2023-10-03T20:25:44.238 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/4212689945 ]] 2023-10-03T20:25:44.238 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/4212689945 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\4\2\1\2\6\8\9\9\4\5 ]] 2023-10-03T20:25:44.238 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:25:45.006 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:45.004+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:25:45.018 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:45.020+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:25:45.032 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:45.032+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:25:48.826 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:48.824+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:25:48.847 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:48.848+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:25:48.848 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:48.848+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:25:51.969 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:51.968+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:25:51.970 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:51.968+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:25:51.970 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:51.968+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:25:51.970 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:25:51.968+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:25:54.251 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:25:54.251 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:25:54.251 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:25:54.252 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:25:54.813 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.808+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:25:54.816 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:25:54.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:25:54.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:25:54.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:25:54.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:25:54.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:25:54.816+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:25:55.640 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:25:55.649 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:25:55.650 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364755 2023-10-03T20:25:55.650 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364755 -le 1696367755 ]] 2023-10-03T20:25:55.650 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/4212689945 ]] 2023-10-03T20:25:55.650 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/4212689945 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\4\2\1\2\6\8\9\9\4\5 ]] 2023-10-03T20:25:55.650 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:26:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.040+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.041 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.040+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1cf7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.040+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.042 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.040+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.048 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.048+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.056+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.056+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.071 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.072+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.071 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.072+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.084+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.086 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.084+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.096+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.096+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.104+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.104+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.104+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.104+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.150 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.151 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.151 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d15080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.151 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d15280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.151 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.148+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d15480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.156 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.156+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:00.157 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:26:00.156+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d15680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:26:02.005 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.004+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:26:02.014 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:02.014 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:26:02.014 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:26:02.014 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:02.014 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:26:02.027 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:02.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:05.652 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:26:05.652 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:26:05.663 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:26:05.680 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:26:05.724 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:05.724+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:05.725 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:05.724+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:26:05.725 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:05.724+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:26:05.725 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:05.724+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:05.725 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:05.724+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:07.050 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:26:07.050 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:26:07.051 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364767 2023-10-03T20:26:07.051 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364767 -le 1696367755 ]] 2023-10-03T20:26:07.051 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/4212689945 ]] 2023-10-03T20:26:07.051 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/4212689945 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\4\2\1\2\6\8\9\9\4\5 ]] 2023-10-03T20:26:07.051 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:26:09.109 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:09.104+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:09.109 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:09.104+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:26:09.109 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:09.104+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:26:09.110 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:09.108+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:26:09.116 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:09.112+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:17.059 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:26:17.059 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:26:17.066 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:26:17.068 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:26:18.516 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/4212689945 2023-10-03T20:26:18.516 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:26:18.517 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364778 2023-10-03T20:26:18.517 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364778 -le 1696367755 ]] 2023-10-03T20:26:18.517 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/4212689945 ]] 2023-10-03T20:26:18.517 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/4212689945 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\4\2\1\2\6\8\9\9\4\5 ]] 2023-10-03T20:26:18.517 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:26:25.422 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:25.420+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:26:25.422 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:25.420+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:26:25.429 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:25.428+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:26:25.429 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:25.428+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:26:28.028 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:28.024+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:28.029 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:28.024+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:28.029 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:28.024+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:28.184 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.184+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:26:28.184 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.184+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:26:28.185 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.184+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:26:28.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.188+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:26:28.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.188+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:26:28.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.188+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:26:28.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.188+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:26:28.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.188+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:26:28.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:28.188+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:26:28.530 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:26:28.530 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:26:28.531 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:26:28.544 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:26:29.849 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:26:29.849 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:26:29.849 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364789 2023-10-03T20:26:29.850 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364789 -le 1696367755 ]] 2023-10-03T20:26:29.850 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2386356822 ]] 2023-10-03T20:26:29.850 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2386356822 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\4\2\1\2\6\8\9\9\4\5 ]] 2023-10-03T20:26:29.850 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2386356822 2023-10-03T20:26:32.378 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2386356822 until 2023-10-03T21:26:31.682696+0000 (3600 sec) 2023-10-03T20:26:32.426 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2386356822 2023-10-03T20:26:32.435 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:26:33.541 INFO:tasks.workunit.client.0.smithi167.stderr:listed 12 entries 2023-10-03T20:26:33.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:26:33.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:26:34.836 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:34.828+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:34.841 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:34.840+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:34.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:34.844+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:26:34.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:34.844+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:35.091 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:35.092+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:26:43.402 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:43.399+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:43.402 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:43.399+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:26:43.402 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:43.399+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:26:43.402 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:43.399+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:43.402 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:43.399+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:26:43.577 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:26:43.577 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:26:43.577 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:26:43.577 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:26:44.666 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:44.663+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:26:44.667 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:26:44.663+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:26:45.522 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:26:45.525 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:26:45.744 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364805 2023-10-03T20:26:45.744 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364805 -le 1696367755 ]] 2023-10-03T20:26:45.744 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2386356822 ]] 2023-10-03T20:26:45.744 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2386356822 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\8\6\3\5\6\8\2\2 ]] 2023-10-03T20:26:45.744 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:26:53.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:53.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:53.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:53.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:26:53.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:53.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:26:53.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:53.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:53.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:53.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:26:53.760 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:53.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:54.574 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:54.571+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:26:54.580 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:54.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:26:54.580 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:54.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:26:54.580 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:54.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:26:54.580 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:54.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:26:54.580 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:26:54.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:26:55.849 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:26:55.851 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:26:55.854 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:26:55.854 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:26:57.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:26:57.111 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:26:57.116 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364817 2023-10-03T20:26:57.116 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364817 -le 1696367755 ]] 2023-10-03T20:26:57.116 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2386356822 ]] 2023-10-03T20:26:57.116 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2386356822 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\8\6\3\5\6\8\2\2 ]] 2023-10-03T20:26:57.117 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:27:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.031+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.031+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.031+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e156d700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.034 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.031+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1264b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.086 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.087 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.083+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.091 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.091 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.091 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.091 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.092 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.092 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.093 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.093 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.093 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.093 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b33080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.094 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.091+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b33880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.095+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.095+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.095+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.095+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.095+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.096 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.095+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.133 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.131+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.133 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.131+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.155 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.156 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.156 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.156 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.156 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.156 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.157 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:00.157 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:27:00.155+0000 7f6ecba9d640 -1 librbd::ImageState: 0x5606e1b32e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:27:07.153 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:27:07.154 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:27:07.154 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:27:07.154 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:27:08.575 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:27:08.578 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:27:08.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364828 2023-10-03T20:27:08.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364828 -le 1696367755 ]] 2023-10-03T20:27:08.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2386356822 ]] 2023-10-03T20:27:08.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2386356822 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\8\6\3\5\6\8\2\2 ]] 2023-10-03T20:27:08.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:27:09.447 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:09.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:27:09.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:09.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:27:09.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:09.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:27:09.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:09.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:27:09.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:09.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:27:09.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:09.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:27:09.737 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:09.735+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:18.618 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:27:18.618 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:27:18.618 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:27:18.644 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:27:19.866 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:27:19.866 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:27:19.867 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364839 2023-10-03T20:27:19.867 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364839 -le 1696367755 ]] 2023-10-03T20:27:19.867 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2386356822 ]] 2023-10-03T20:27:19.868 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2386356822 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\8\6\3\5\6\8\2\2 ]] 2023-10-03T20:27:19.868 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:27:21.168 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:21.163+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:27:21.203 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:21.163+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:27:21.204 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:21.171+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:21.204 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:21.179+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:27:21.204 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:21.179+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:27:21.204 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:21.179+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:27:24.582 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:24.579+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:24.592 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:24.591+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:27:24.856 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:24.851+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:27:24.864 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:24.863+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:27:24.864 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:24.863+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:27:25.588 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:25.586+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:25.951 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.950+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:27:25.965 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.962+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:27:25.965 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.962+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:27:25.965 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.962+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:27:25.965 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.962+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:27:25.965 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.962+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:27:25.966 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:25.962+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:27:29.873 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:27:29.891 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:27:29.892 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:27:29.904 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:27:31.166 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2386356822 2023-10-03T20:27:31.175 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:27:31.457 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364851 2023-10-03T20:27:31.457 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364851 -le 1696367755 ]] 2023-10-03T20:27:31.457 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2386356822 ]] 2023-10-03T20:27:31.457 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2386356822 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\8\6\3\5\6\8\2\2 ]] 2023-10-03T20:27:31.457 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:27:31.703 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:31.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:27:31.708 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:31.706+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:27:37.251 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:37.250+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:37.274 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:37.270+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:27:41.465 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:27:41.466 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:27:41.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:27:41.476 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:27:42.868 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:27:42.868 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:27:42.869 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364862 2023-10-03T20:27:42.869 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364862 -le 1696367755 ]] 2023-10-03T20:27:42.869 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:27:42.869 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\8\6\3\5\6\8\2\2 ]] 2023-10-03T20:27:42.869 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/344627823 2023-10-03T20:27:42.913 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:42.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:27:42.922 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:42.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:27:42.922 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:42.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:27:44.581 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:44.578+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:44.608 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:44.606+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:27:45.415 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/344627823 until 2023-10-03T21:27:44.623118+0000 (3600 sec) 2023-10-03T20:27:45.481 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/344627823 2023-10-03T20:27:45.481 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:27:46.070 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:46.066+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:27:46.079 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:46.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:27:46.079 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:46.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:27:46.425 INFO:tasks.workunit.client.0.smithi167.stderr:listed 13 entries 2023-10-03T20:27:46.454 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:27:46.454 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:27:51.756 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:51.754+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:27:51.756 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:51.754+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:27:51.779 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:27:51.774+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:27:54.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:54.126+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:27:55.776 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:55.774+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:27:55.776 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:55.774+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:27:55.776 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:55.774+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:27:55.783 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:55.778+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:27:55.784 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:27:55.778+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:27:56.472 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:27:56.472 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:27:56.476 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:27:56.483 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:27:58.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:27:58.113 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:27:58.114 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364878 2023-10-03T20:27:58.114 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364878 -le 1696367755 ]] 2023-10-03T20:27:58.114 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:27:58.114 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:27:58.114 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:28:00.013 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.010+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.013 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.010+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.013 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.010+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d15e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.013 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.010+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e159a800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.018 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.014+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.018 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.014+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.018 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.014+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.019 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.014+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1d14400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.030+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.030+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.030+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e0d5fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.030+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1bfc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.054+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.054+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c90600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.066 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.062+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.066 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.062+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.066 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.062+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1cf7100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.066 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.062+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b32180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.091 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.086+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.091 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.086+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b32680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.098+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.098+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e122ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1c91480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b33280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b33500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b33700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.106 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.102+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b33a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b33c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1b33e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1ece000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:00.108 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:28:00.106+0000 7f6ecc29e640 -1 librbd::ImageState: 0x5606e1ece200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:28:02.473 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:02.470+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:04.831 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:04.830+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:28:04.831 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:04.830+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:28:04.839 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:04.838+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:28:04.839 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:04.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:28:04.839 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:04.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:28:05.565 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:28:05.565 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:28:05.565 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:28:05.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:28:05.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:28:05.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:28:05.567 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:05.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:28:08.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:28:08.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:28:08.130 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:28:08.137 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:28:09.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:28:09.547 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:28:09.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364889 2023-10-03T20:28:09.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364889 -le 1696367755 ]] 2023-10-03T20:28:09.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:28:09.549 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:28:09.549 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:28:12.210 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:12.209+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:28:12.211 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:12.209+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:28:12.216 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:12.217+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:12.216 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:12.217+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:28:12.621 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:12.621+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:12.648 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:12.649+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:28:14.475 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:28:14.475 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:28:14.476 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:28:14.476 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:28:14.476 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:28:14.476 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:28:14.476 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:14.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:28:18.664 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:18.665+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:19.033 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:19.029+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:28:19.034 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:19.029+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:28:19.034 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:19.029+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:28:19.034 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:19.033+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:28:19.550 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:28:19.551 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:28:19.551 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:28:19.552 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:28:20.548 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:20.545+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:28:20.548 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:20.545+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:28:20.548 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:20.549+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:20.558 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:20.557+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:28:20.559 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:20.557+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:28:20.559 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:20.557+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:28:20.810 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:28:20.810 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:28:20.811 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364900 2023-10-03T20:28:20.811 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364900 -le 1696367755 ]] 2023-10-03T20:28:20.811 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:28:20.811 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:28:20.811 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:28:21.456 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:21.457+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:30.819 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:28:30.819 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:28:30.819 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:28:30.819 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:28:32.105 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:28:32.112 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:28:32.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364912 2023-10-03T20:28:32.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364912 -le 1696367755 ]] 2023-10-03T20:28:32.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:28:32.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:28:32.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:28:32.160 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:32.161+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:28:32.161 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:32.161+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:28:32.166 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:32.165+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:32.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:32.165+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:28:32.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:32.165+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:28:32.173 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:32.173+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:28:34.055 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:34.053+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:28:34.055 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:34.053+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:28:34.060 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:34.057+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:28:34.065 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:34.061+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:28:34.066 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:34.061+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:28:35.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:35.417+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:28:35.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:35.417+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:28:35.423 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:35.425+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:35.425 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:35.425+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:28:35.428 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:35.429+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:28:42.113 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:28:42.120 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:28:42.123 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:28:42.123 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:28:42.852 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:42.853+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:28:42.853 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:42.853+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:28:42.854 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:42.853+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:28:42.864 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:42.865+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:28:42.864 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:28:42.865+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:28:43.544 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:28:43.544 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:28:43.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364923 2023-10-03T20:28:43.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364923 -le 1696367755 ]] 2023-10-03T20:28:43.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:28:43.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:28:43.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:28:53.547 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:28:53.552 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:28:53.552 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:28:53.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:28:55.012 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/344627823 2023-10-03T20:28:55.012 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:28:55.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364935 2023-10-03T20:28:55.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364935 -le 1696367755 ]] 2023-10-03T20:28:55.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/344627823 ]] 2023-10-03T20:28:55.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/344627823 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:28:55.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:28:55.864 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:55.857+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:28:55.864 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:55.857+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:28:55.875 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:55.873+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:28:55.878 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:55.873+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:28:55.878 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:28:55.877+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:29:05.016 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:29:05.016 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:29:05.016 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:29:05.032 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:29:05.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:29:05.957 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:29:05.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364945 2023-10-03T20:29:05.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364945 -le 1696367755 ]] 2023-10-03T20:29:05.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/584508052 ]] 2023-10-03T20:29:05.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/584508052 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\3\4\4\6\2\7\8\2\3 ]] 2023-10-03T20:29:05.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/584508052 2023-10-03T20:29:06.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:06.048+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:29:06.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:06.048+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:29:06.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:06.048+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:29:06.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:06.052+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:29:06.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:06.052+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:29:06.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:06.056+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:29:08.459 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/584508052 until 2023-10-03T21:29:07.644235+0000 (3600 sec) 2023-10-03T20:29:08.499 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:29:08.500 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/584508052 2023-10-03T20:29:09.284 INFO:tasks.workunit.client.0.smithi167.stderr:listed 14 entries 2023-10-03T20:29:09.319 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:29:09.320 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:29:11.671 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:11.668+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:29:19.321 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:29:19.323 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:29:19.326 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:29:19.334 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:29:20.493 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:29:20.493 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:29:20.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364960 2023-10-03T20:29:20.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364960 -le 1696367755 ]] 2023-10-03T20:29:20.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/584508052 ]] 2023-10-03T20:29:20.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/584508052 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\8\4\5\0\8\0\5\2 ]] 2023-10-03T20:29:20.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:29:22.292 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:22.292+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:29:22.292 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:22.292+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:29:22.299 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:22.296+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:29:22.306 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:22.304+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:29:22.306 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:22.304+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:29:30.503 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:29:30.514 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:29:30.520 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:29:30.525 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:29:31.766 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:29:31.766 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:29:31.767 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364971 2023-10-03T20:29:31.767 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364971 -le 1696367755 ]] 2023-10-03T20:29:31.768 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/584508052 ]] 2023-10-03T20:29:31.768 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/584508052 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\8\4\5\0\8\0\5\2 ]] 2023-10-03T20:29:31.768 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:29:41.769 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:29:41.770 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:29:41.771 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:29:41.782 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:29:42.924 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:29:42.936 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:29:42.938 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364982 2023-10-03T20:29:42.938 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364982 -le 1696367755 ]] 2023-10-03T20:29:42.938 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/584508052 ]] 2023-10-03T20:29:42.938 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/584508052 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\8\4\5\0\8\0\5\2 ]] 2023-10-03T20:29:42.938 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:29:49.897 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:49.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:29:49.898 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:49.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:29:49.898 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:49.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:29:52.941 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:29:52.962 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:29:52.966 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:29:52.966 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:29:53.712 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:29:53.712 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:29:53.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:29:53.712+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:29:54.375 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:29:54.375 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:29:54.381 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696364994 2023-10-03T20:29:54.382 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696364994 -le 1696367755 ]] 2023-10-03T20:29:54.382 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/584508052 ]] 2023-10-03T20:29:54.382 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/584508052 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\8\4\5\0\8\0\5\2 ]] 2023-10-03T20:29:54.382 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:29:55.069 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.068+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:29:55.079 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:29:55.080 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:29:55.080 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:29:55.080 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:29:55.080 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:29:55.080 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:55.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:29:56.866 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:56.863+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:29:56.873 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:56.871+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:29:56.873 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:56.871+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:29:56.873 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:56.871+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:29:56.876 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:56.875+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:29:56.876 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:29:56.875+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:30:00.053 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:00.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:30:00.056 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:00.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:30:00.056 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:00.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:30:00.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:00.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:30:00.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:00.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:30:00.057 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:00.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:30:01.658 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:30:01.658 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.655+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:30:01.658 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.655+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:30:01.671 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.671+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:30:01.672 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.671+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:30:01.672 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.671+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:30:01.672 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:01.671+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:30:03.141 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:03.139+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:04.401 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:30:04.401 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:30:04.403 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:30:04.403 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:30:05.826 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/584508052 2023-10-03T20:30:05.831 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:30:05.833 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365005 2023-10-03T20:30:05.834 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365005 -le 1696367755 ]] 2023-10-03T20:30:05.834 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/584508052 ]] 2023-10-03T20:30:05.834 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/584508052 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\8\4\5\0\8\0\5\2 ]] 2023-10-03T20:30:05.834 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:30:06.784 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:06.783+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:06.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:06.803+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:30:14.208 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:14.207+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:30:14.396 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:14.395+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:30:14.403 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:14.399+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:30:14.412 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:14.403+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:30:15.847 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:30:15.847 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:30:15.849 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:30:15.863 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:30:17.295 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:30:17.296 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:30:17.296 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365017 2023-10-03T20:30:17.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365017 -le 1696367755 ]] 2023-10-03T20:30:17.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1068382668 ]] 2023-10-03T20:30:17.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1068382668 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\5\8\4\5\0\8\0\5\2 ]] 2023-10-03T20:30:17.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/1068382668 2023-10-03T20:30:17.854 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:17.851+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:17.917 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:17.911+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:30:17.917 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:17.911+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:30:17.917 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:17.911+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:30:19.448 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/1068382668 until 2023-10-03T21:30:18.787237+0000 (3600 sec) 2023-10-03T20:30:19.491 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/1068382668 2023-10-03T20:30:19.492 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:30:20.427 INFO:tasks.workunit.client.0.smithi167.stderr:listed 15 entries 2023-10-03T20:30:20.488 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:30:20.488 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:30:26.750 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:26.747+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:30:26.751 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:26.747+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:30:26.752 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:26.751+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:26.775 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:26.775+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:30:26.775 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:26.775+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:30:26.775 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:26.775+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:30:28.131 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:28.127+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:28.154 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:28.151+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:30:30.494 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:30:30.494 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:30:30.495 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:30:30.501 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:30:31.713 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:30:31.714 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:30:31.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365031 2023-10-03T20:30:31.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365031 -le 1696367755 ]] 2023-10-03T20:30:31.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1068382668 ]] 2023-10-03T20:30:31.714 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1068382668 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\0\6\8\3\8\2\6\6\8 ]] 2023-10-03T20:30:31.715 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:30:34.747 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:34.739+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:30:34.747 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:34.739+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:30:34.747 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:34.739+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:30:41.719 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:30:41.731 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:30:41.734 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:30:41.744 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:30:42.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:30:42.958 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:30:42.958 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365042 2023-10-03T20:30:42.959 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365042 -le 1696367755 ]] 2023-10-03T20:30:42.959 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1068382668 ]] 2023-10-03T20:30:42.959 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1068382668 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\0\6\8\3\8\2\6\6\8 ]] 2023-10-03T20:30:42.959 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:30:47.337 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.331+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:30:47.344 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.343+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:30:47.368 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.363+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:30:47.559 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.547+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:30:47.563 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:30:47.563 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:30:47.563 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:30:47.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:30:48.179 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:48.175+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:49.726 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:49.723+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:30:49.738 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:49.735+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:30:49.744 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:30:49.743+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:30:52.960 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:30:52.960 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:30:52.962 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:30:52.965 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:30:54.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:30:54.172 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:30:54.172 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365054 2023-10-03T20:30:54.173 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365054 -le 1696367755 ]] 2023-10-03T20:30:54.173 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1068382668 ]] 2023-10-03T20:30:54.173 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1068382668 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\0\6\8\3\8\2\6\6\8 ]] 2023-10-03T20:30:54.173 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:31:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.046+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.046+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.046+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1c63e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.051 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.046+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d15380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.056 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.056 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.056 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d41780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.056 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d15880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1cf6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ece500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1eced00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.057 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecfb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecf780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1c63e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.067 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.066+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.067 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.066+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d15380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.128 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.128 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.128 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d41780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d15880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1cf6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ece500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1eced00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.129 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.130 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecfb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:00.130 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:00.126+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecf780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:04.177 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:31:04.179 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:31:04.180 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:31:04.196 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:31:05.684 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:31:05.685 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:31:05.685 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365065 2023-10-03T20:31:05.686 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365065 -le 1696367755 ]] 2023-10-03T20:31:05.686 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1068382668 ]] 2023-10-03T20:31:05.686 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1068382668 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\0\6\8\3\8\2\6\6\8 ]] 2023-10-03T20:31:05.686 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:31:15.214 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.210+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:31:15.223 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.214+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:31:15.251 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.246+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:31:15.252 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.246+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:31:15.687 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:31:15.688 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:31:15.688 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:31:15.688 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:31:15.793 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.790+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:31:15.811 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.806+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:31:15.815 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.814+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:31:15.832 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:15.830+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:31:17.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/1068382668 2023-10-03T20:31:17.015 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:31:17.019 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365077 2023-10-03T20:31:17.019 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365077 -le 1696367755 ]] 2023-10-03T20:31:17.019 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/1068382668 ]] 2023-10-03T20:31:17.019 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/1068382668 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\0\6\8\3\8\2\6\6\8 ]] 2023-10-03T20:31:17.019 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:31:20.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:20.478+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:31:20.488 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:20.482+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:31:20.490 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:20.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:31:20.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:20.494+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:31:20.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:20.494+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:31:21.606 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:21.602+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:31:21.629 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:21.626+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:31:21.631 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:21.630+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:31:21.631 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:21.630+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:31:21.632 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:21.630+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:31:24.695 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:24.694+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:31:24.701 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:24.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:31:24.707 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:24.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:31:27.052 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:31:27.054 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:31:27.056 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:31:27.059 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:31:28.608 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2447272347 2023-10-03T20:31:28.625 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:31:28.627 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365088 2023-10-03T20:31:28.627 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365088 -le 1696367755 ]] 2023-10-03T20:31:28.628 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2447272347 ]] 2023-10-03T20:31:28.628 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2447272347 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\1\0\6\8\3\8\2\6\6\8 ]] 2023-10-03T20:31:28.628 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2447272347 2023-10-03T20:31:29.481 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:31:29.478+0000 7f6ec9298640 -1 librbd::api::Namespace: list: error listing namespaces: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:31:30.063 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:30.062+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:31:30.469 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2447272347 until 2023-10-03T21:31:29.781853+0000 (3600 sec) 2023-10-03T20:31:30.534 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:31:30.534 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2447272347 2023-10-03T20:31:31.960 INFO:tasks.workunit.client.0.smithi167.stderr:listed 16 entries 2023-10-03T20:31:32.033 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2447272347 2023-10-03T20:31:32.033 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:31:39.662 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:39.658+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:31:39.662 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:39.658+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:31:39.672 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:39.670+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:31:39.673 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:31:39.670+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:31:42.037 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:31:42.039 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:31:42.046 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:31:42.055 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:31:43.233 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:31:43.234 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:31:43.235 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365103 2023-10-03T20:31:43.235 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365103 -le 1696367755 ]] 2023-10-03T20:31:43.235 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:31:43.235 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\4\4\7\2\7\2\3\4\7 ]] 2023-10-03T20:31:43.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist add 172.21.15.167:0/2335679350 2023-10-03T20:31:45.528 INFO:tasks.workunit.client.0.smithi167.stderr:blocklisting 172.21.15.167:0/2335679350 until 2023-10-03T21:31:44.881029+0000 (3600 sec) 2023-10-03T20:31:45.592 INFO:tasks.workunit.client.0.smithi167.stderr:+ ceph osd blocklist ls 2023-10-03T20:31:45.599 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep -q 172.21.15.167:0/2335679350 2023-10-03T20:31:46.675 INFO:tasks.workunit.client.0.smithi167.stderr:listed 17 entries 2023-10-03T20:31:46.733 INFO:tasks.workunit.client.0.smithi167.stderr:+ PREV_CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:31:46.734 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:31:47.467 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:47.470+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:31:47.467 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:47.470+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:31:47.467 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:47.470+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:31:47.468 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:31:47.470+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:31:56.735 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:31:56.750 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:31:56.755 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:31:56.764 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:31:58.036 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:31:58.036 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:31:58.037 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365118 2023-10-03T20:31:58.037 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365118 -le 1696367755 ]] 2023-10-03T20:31:58.038 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:31:58.038 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:31:58.038 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:32:00.002 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.002+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.002 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.002+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e0b45800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.005 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.006+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.005 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.006+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1cf7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.009 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.010+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.009 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.010+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.018+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.016 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.018+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1eceb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.021 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.022+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.021 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.022+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecf900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.027 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.026+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.027 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.026+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ecea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.031 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.034+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.031 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.034+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1b32a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.043 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.042+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.043 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.042+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ece300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.046+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.047 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.046+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1b33000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.050+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.050+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1b33b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.053 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.054+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.053 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.054+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1b32600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.058+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.058+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ece500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.062+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.062+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1d15600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.073 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.074+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.075 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.078+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1bfd300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.078 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.078+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.078 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.078+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1b32800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.082 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.082+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.082 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.082+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ebe880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.088 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.090+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.088 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.090+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ebf380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.102+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.099 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.102+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ebf580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.110+0000 7f6ecca9f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:00.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T20:32:00.110+0000 7f6ecca9f640 -1 librbd::ImageState: 0x5606e1ebfe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T20:32:03.133 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:03.133+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:32:03.145 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:03.141+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:32:03.155 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:03.157+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:32:05.778 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:05.777+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:32:06.134 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:32:06.138 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:32:06.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.149+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:32:06.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.149+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:32:06.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.149+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:32:06.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.149+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:32:06.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:06.149+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:32:08.039 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:32:08.042 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:32:08.042 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:32:08.047 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:32:09.203 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:32:09.203 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:32:09.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365129 2023-10-03T20:32:09.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365129 -le 1696367755 ]] 2023-10-03T20:32:09.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:32:09.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:32:09.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:32:19.208 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:32:19.208 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:32:19.211 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:32:19.217 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:32:20.583 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:32:20.583 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:32:20.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365140 2023-10-03T20:32:20.593 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365140 -le 1696367755 ]] 2023-10-03T20:32:20.594 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:32:20.594 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:32:20.594 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:32:30.596 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:32:30.596 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:32:30.596 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:32:30.623 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:32:32.202 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:32:32.203 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:32:32.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365152 2023-10-03T20:32:32.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365152 -le 1696367755 ]] 2023-10-03T20:32:32.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:32:32.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:32:32.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:32:36.206 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:36.205+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:32:36.210 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:36.209+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:32:38.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.749+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:32:38.758 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:32:38.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:32:38.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:32:38.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:32:38.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:32:38.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:32:38.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:32:42.222 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:32:42.227 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:32:42.229 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:32:42.236 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:32:43.647 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:43.649+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:32:43.681 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:32:43.687 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:32:43.689 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365163 2023-10-03T20:32:43.689 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365163 -le 1696367755 ]] 2023-10-03T20:32:43.689 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:32:43.690 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:32:43.690 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:32:45.083 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:45.081+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:32:45.107 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:32:45.097+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:32:53.704 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:32:53.704 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:32:53.708 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:32:53.714 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:32:54.992 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:32:55.003 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:32:55.006 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365175 2023-10-03T20:32:55.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365175 -le 1696367755 ]] 2023-10-03T20:32:55.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:32:55.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:32:55.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:33:05.018 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:33:05.023 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:33:05.033 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:33:05.033 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:33:06.401 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:33:06.402 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:33:06.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365186 2023-10-03T20:33:06.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365186 -le 1696367755 ]] 2023-10-03T20:33:06.403 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:33:06.403 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:33:06.403 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:33:09.261 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:09.261+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:11.739 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:11.740+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:33:11.739 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:11.740+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:33:11.748 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:11.744+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:11.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:11.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:33:11.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:11.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:33:11.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:11.768+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:33:12.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.252+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:33:12.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.252+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:33:12.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.252+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:33:12.282 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:33:12.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:12.276+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:33:16.406 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:33:16.416 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:33:16.416 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:33:16.426 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:33:17.715 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:33:17.719 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:33:17.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365197 2023-10-03T20:33:17.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365197 -le 1696367755 ]] 2023-10-03T20:33:17.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:33:17.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:33:17.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:33:23.483 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.484+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:33:23.488 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.488+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:23.494 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.496+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:33:23.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.500+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:33:23.519 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.520+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:33:23.531 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.532+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:33:23.531 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:23.532+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:33:25.454 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:25.448+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:33:25.468 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:25.464+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:25.471 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:25.472+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:33:25.477 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:25.476+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:33:25.483 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:25.480+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:33:27.730 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:33:27.736 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:33:27.745 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:33:27.756 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:33:29.069 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:33:29.069 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:33:29.070 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365209 2023-10-03T20:33:29.070 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365209 -le 1696367755 ]] 2023-10-03T20:33:29.070 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:33:29.070 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:33:29.070 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:33:34.894 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:34.892+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:33:34.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:34.912+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:33:34.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:34.912+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:34.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:34.912+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:33:34.914 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:34.912+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:33:34.914 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:34.912+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:33:39.087 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:33:39.090 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:33:39.090 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:33:39.091 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:33:40.543 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:33:40.546 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:33:40.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365220 2023-10-03T20:33:40.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365220 -le 1696367755 ]] 2023-10-03T20:33:40.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:33:40.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:33:40.548 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:33:41.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:41.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:33:41.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:41.356+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:33:41.364 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:41.364+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:33:41.412 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:41.412+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:33:41.413 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:33:41.412+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:33:45.122 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:45.120+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:33:50.549 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:33:50.554 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:33:50.557 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:33:50.558 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:33:51.787 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:33:51.787 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:33:51.788 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365231 2023-10-03T20:33:51.788 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365231 -le 1696367755 ]] 2023-10-03T20:33:51.788 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:33:51.788 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:33:51.788 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:33:52.590 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:52.584+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:33:52.591 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:52.592+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:33:52.591 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:52.592+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:33:52.596 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:52.596+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:33:52.602 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:52.600+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:33:54.535 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:54.536+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:33:54.543 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:54.540+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:33:54.553 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:33:54.552+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:01.795 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:34:01.795 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:34:01.795 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:34:01.814 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:34:02.891 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:34:02.891 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:34:02.892 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365242 2023-10-03T20:34:02.892 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365242 -le 1696367755 ]] 2023-10-03T20:34:02.892 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:34:02.892 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:34:02.892 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:34:03.129 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:03.128+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:34:03.129 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:03.128+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:34:07.841 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:07.836+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:07.842 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:07.836+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:07.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:07.844+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:12.120 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:12.115+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:12.894 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:34:12.895 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:34:12.910 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:34:12.914 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:34:14.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:34:14.182 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:34:14.190 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365254 2023-10-03T20:34:14.190 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365254 -le 1696367755 ]] 2023-10-03T20:34:14.191 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:34:14.191 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:34:14.191 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:34:16.915 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:16.915+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:34:16.933 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:16.931+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:34:17.302 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:17.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:17.302 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:17.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:34:17.302 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:17.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:17.302 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:17.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:17.478 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:17.475+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:34:17.478 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:17.475+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:34:17.478 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:17.475+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:22.068 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:22.067+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:22.078 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:22.075+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:34:22.078 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:22.075+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:24.197 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:34:24.197 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:34:24.203 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:34:24.216 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:34:24.487 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:24.483+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:24.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:24.499+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:34:24.596 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:24.595+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:24.610 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:24.603+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:24.628 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:24.615+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:34:24.628 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:24.615+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:25.142 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:25.139+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:34:25.142 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:25.139+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:34:25.158 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:25.155+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:25.163 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:25.159+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:34:25.163 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:25.159+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:34:25.176 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:25.175+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:34:25.554 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:34:25.555 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:34:25.556 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365265 2023-10-03T20:34:25.556 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365265 -le 1696367755 ]] 2023-10-03T20:34:25.556 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:34:25.556 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:34:25.556 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:34:29.276 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:29.275+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:29.276 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:29.275+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:34:29.276 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:29.275+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:34:29.276 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:29.275+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:34:35.479 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:34:35.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.487+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:35.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.487+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:34:35.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.487+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:34:35.489 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.487+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:35.494 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.491+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:34:35.494 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:35.491+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:35.558 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:34:35.559 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:34:35.559 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:34:35.580 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:34:36.937 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:34:36.938 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:34:36.939 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365276 2023-10-03T20:34:36.940 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365276 -le 1696367755 ]] 2023-10-03T20:34:36.940 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:34:36.940 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:34:36.940 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:34:37.311 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:37.307+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:37.326 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:37.323+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:34:40.892 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:40.891+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:40.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:40.911+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:34:41.722 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:41.718+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:46.941 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:34:46.942 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:34:46.943 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:34:46.947 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:34:48.380 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:34:48.383 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:34:48.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365288 2023-10-03T20:34:48.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365288 -le 1696367755 ]] 2023-10-03T20:34:48.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:34:48.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:34:48.384 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:34:50.062 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:50.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:34:50.065 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:50.062+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:50.065 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:50.062+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:34:50.080 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:50.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:50.107 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:50.102+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:34:50.107 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:50.102+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:55.554 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:34:55.567 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:34:55.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:34:55.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:34:55.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:34:55.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:34:55.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:34:55.566+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:34:56.937 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:56.926+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:34:56.937 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:56.926+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:34:56.937 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:34:56.926+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:34:58.390 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:34:58.392 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:34:58.400 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:34:58.405 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:34:59.680 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:34:59.685 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:34:59.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365299 2023-10-03T20:34:59.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365299 -le 1696367755 ]] 2023-10-03T20:34:59.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:34:59.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:34:59.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:35:03.361 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:03.358+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:35:03.361 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:03.358+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:35:03.361 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:03.358+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:35:03.371 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:03.362+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:35:03.371 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:03.362+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:35:03.371 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:03.362+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:35:04.288 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:04.286+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:35:04.288 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:04.286+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:35:04.294 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:04.290+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:35:04.299 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:04.290+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:35:04.300 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:04.298+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:35:04.311 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:04.310+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:35:09.702 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:35:09.702 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:35:09.702 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:35:09.703 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:35:11.005 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:35:11.007 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:35:11.015 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365311 2023-10-03T20:35:11.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365311 -le 1696367755 ]] 2023-10-03T20:35:11.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:35:11.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:35:11.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:35:12.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:12.998+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:35:12.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:12.998+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:35:13.009 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:13.006+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:35:13.044 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:13.042+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:35:13.044 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:13.042+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:35:13.044 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:13.042+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:35:13.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:13.042+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:35:13.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:13.042+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:35:21.025 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:35:21.025 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:35:21.025 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:35:21.025 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:35:21.548 INFO:tasks.workunit.client.0.smithi167.stderr:2023-10-03T20:35:21.549+0000 7f52f0f82640 0 --2- 172.21.15.167:0/305001256 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52ec0719c0 0x7f52ec123250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-03T20:35:22.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:35:22.494 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:35:22.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365322 2023-10-03T20:35:22.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365322 -le 1696367755 ]] 2023-10-03T20:35:22.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:35:22.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:35:22.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:35:23.538 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.537+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:35:23.539 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.537+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:35:23.539 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.537+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:35:23.540 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.537+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:35:23.540 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.537+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:35:23.540 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.537+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:35:23.540 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:23.541+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:35:29.131 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:29.129+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:35:29.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:29.149+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:35:29.215 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:29.213+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:35:29.220 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:29.221+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:35:29.226 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:35:29.225+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:35:32.496 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:35:32.497 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:35:32.497 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:35:32.499 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:35:33.947 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:35:33.947 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:35:33.949 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365333 2023-10-03T20:35:33.949 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365333 -le 1696367755 ]] 2023-10-03T20:35:33.950 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:35:33.950 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:35:33.950 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:35:43.952 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:35:43.957 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:35:43.958 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:35:43.982 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:35:45.267 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:35:45.267 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:35:45.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365345 2023-10-03T20:35:45.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365345 -le 1696367755 ]] 2023-10-03T20:35:45.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:35:45.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:35:45.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:35:49.829 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:49.820+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:35:49.829 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:49.820+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:35:49.839 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:49.836+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:35:49.851 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:49.848+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:35:49.851 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:49.848+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:35:49.852 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:35:49.852+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:35:55.282 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:35:55.282 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:35:55.283 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:35:55.292 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:35:56.272 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:35:56.290 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:35:56.291 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365356 2023-10-03T20:35:56.292 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365356 -le 1696367755 ]] 2023-10-03T20:35:56.292 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:35:56.292 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:35:56.292 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:36:03.237 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:03.236+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:36:03.298 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:03.296+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:36:06.304 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:36:06.307 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:36:06.307 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:36:06.311 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:36:06.389 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.380+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:36:06.389 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.384+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:06.644 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.636+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:36:06.645 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.636+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:36:06.653 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.652+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:06.663 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.660+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:36:06.663 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:06.660+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:36:07.442 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:36:07.443 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:36:07.443 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365367 2023-10-03T20:36:07.443 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365367 -le 1696367755 ]] 2023-10-03T20:36:07.444 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:36:07.444 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:36:07.444 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:36:09.173 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:09.172+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:09.183 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:09.180+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:36:16.676 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:16.672+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:17.446 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:36:17.448 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:36:17.475 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:36:17.476 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:36:18.617 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:36:18.623 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:36:18.629 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365378 2023-10-03T20:36:18.629 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365378 -le 1696367755 ]] 2023-10-03T20:36:18.630 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:36:18.630 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:36:18.630 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:36:22.087 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:22.079+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:36:23.963 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:23.955+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:36:23.963 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:23.955+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:36:23.963 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:23.955+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:28.636 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:36:28.636 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:36:28.640 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:36:28.649 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:36:29.853 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:36:29.861 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:36:29.862 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365389 2023-10-03T20:36:29.862 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365389 -le 1696367755 ]] 2023-10-03T20:36:29.862 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:36:29.862 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:36:29.862 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:36:30.597 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:30.595+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:34.081 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:34.075+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:34.086 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:34.075+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:36:39.877 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:36:39.877 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:36:39.878 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:36:39.879 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:36:40.615 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:40.611+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:36:40.628 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:40.627+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:36:41.163 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:36:41.163 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:36:41.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365401 2023-10-03T20:36:41.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365401 -le 1696367755 ]] 2023-10-03T20:36:41.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:36:41.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:36:41.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:36:46.250 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:46.247+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:36:46.252 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:46.247+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:36:46.252 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:46.247+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:36:46.260 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:46.259+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:36:46.284 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:46.279+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:36:46.285 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:46.279+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:36:48.131 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:48.127+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:48.157 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:48.151+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:36:48.342 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:48.331+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:36:48.343 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:48.339+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:36:51.181 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:36:51.186 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:36:51.197 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:36:51.215 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:36:52.633 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:36:52.640 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:36:52.645 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365412 2023-10-03T20:36:52.645 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365412 -le 1696367755 ]] 2023-10-03T20:36:52.645 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:36:52.645 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:36:52.646 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:36:52.860 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:36:52.855+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:36:57.742 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:57.738+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:36:57.754 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:36:57.750+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:37:00.527 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:00.522+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:00.527 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:00.522+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:01.529 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:01.526+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:02.110 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:02.106+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:37:02.651 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:37:02.651 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:37:02.655 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:37:02.655 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:37:04.105 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:37:04.109 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:37:04.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365424 2023-10-03T20:37:04.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365424 -le 1696367755 ]] 2023-10-03T20:37:04.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:37:04.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:37:04.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:37:05.334 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:05.330+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:37:05.339 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:05.334+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:37:05.350 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:05.346+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:37:05.351 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:05.346+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:37:08.256 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:08.250+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:08.256 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:08.250+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:08.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:08.254+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:08.264 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:08.258+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:09.619 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:09.610+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:09.620 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:09.610+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:09.620 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:09.614+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:09.625 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:09.618+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:09.625 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:09.618+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:09.649 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:09.638+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:37:11.822 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:11.818+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:11.822 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:11.818+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:11.822 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:11.818+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:11.822 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:11.818+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:14.112 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:37:14.112 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:37:14.115 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:37:14.123 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:37:15.532 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:37:15.533 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:37:15.533 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365435 2023-10-03T20:37:15.533 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365435 -le 1696367755 ]] 2023-10-03T20:37:15.533 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:37:15.534 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:37:15.534 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:37:21.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:21.710+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:21.713 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:21.710+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:21.716 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:21.710+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:21.719 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:21.714+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:21.763 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:21.758+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:37:25.535 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:37:25.536 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:37:25.540 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:37:25.544 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:37:26.798 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:37:26.799 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:37:26.799 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365446 2023-10-03T20:37:26.799 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365446 -le 1696367755 ]] 2023-10-03T20:37:26.800 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:37:26.800 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:37:26.800 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:37:27.160 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:27.158+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:27.160 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:27.158+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:27.169 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:27.166+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:27.177 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:27.174+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:27.177 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:27.174+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:27.757 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:27.754+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:37:27.760 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:27.758+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:37:28.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:37:28.486 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:37:28.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:37:28.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:37:28.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:37:28.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:37:28.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:28.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:37:30.903 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:30.889+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:30.907 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:30.889+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:30.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:30.901+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:30.929 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:30.921+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:30.929 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:30.925+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:30.934 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:30.933+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:37:36.036 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:36.033+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:36.074 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:36.073+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:36.074 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:36.073+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:36.079 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:36.077+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:37:36.802 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:37:36.802 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:37:36.808 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:37:36.814 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:37:38.028 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:37:38.028 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:37:38.029 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365458 2023-10-03T20:37:38.030 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365458 -le 1696367755 ]] 2023-10-03T20:37:38.030 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:37:38.030 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:37:38.030 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:37:45.699 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:45.697+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:45.702 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:45.701+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:45.725 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:45.713+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:48.039 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:37:48.040 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:37:48.046 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:37:48.055 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:37:49.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:37:49.260 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:37:49.260 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365469 2023-10-03T20:37:49.260 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365469 -le 1696367755 ]] 2023-10-03T20:37:49.260 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:37:49.260 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:37:49.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:37:50.962 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:50.957+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:50.976 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:50.969+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:37:51.346 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:51.341+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:51.346 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:51.341+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:51.346 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:51.341+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:51.346 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:51.341+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:56.168 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.165+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:56.181 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.177+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:37:56.181 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.177+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:37:56.189 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.185+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:37:56.969 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.965+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:37:56.969 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.965+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:37:56.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.973+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:37:56.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.973+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:37:56.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:37:56.973+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:37:57.446 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:57.445+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:37:57.446 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:57.445+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:37:57.447 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:57.445+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:37:57.466 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:57.465+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:37:58.382 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:58.377+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:37:58.395 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:37:58.393+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:37:59.262 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:37:59.273 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:37:59.273 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:37:59.277 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:38:00.776 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:38:00.776 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:38:00.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365480 2023-10-03T20:38:00.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365480 -le 1696367755 ]] 2023-10-03T20:38:00.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:38:00.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:38:00.777 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:38:10.797 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:38:10.805 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:38:10.809 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:38:10.814 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:38:11.981 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:38:11.993 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:38:12.000 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365492 2023-10-03T20:38:12.000 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365492 -le 1696367755 ]] 2023-10-03T20:38:12.000 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:38:12.000 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:38:12.000 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:38:12.536 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:12.528+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:38:13.103 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.100+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:38:13.127 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.124+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:38:13.128 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.124+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:38:13.128 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.124+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:38:13.140 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:38:13.140 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:38:13.141 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:13.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:38:18.336 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:18.328+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:38:18.345 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:18.340+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:38:22.026 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:38:22.027 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:38:22.027 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:38:22.027 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:38:23.406 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:38:23.406 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:38:23.407 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365503 2023-10-03T20:38:23.407 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365503 -le 1696367755 ]] 2023-10-03T20:38:23.407 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:38:23.407 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:38:23.407 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:38:29.866 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:29.860+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:38:29.889 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:29.884+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:38:30.126 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:30.120+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:38:30.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:30.120+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:38:30.132 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:30.128+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:38:30.139 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:30.136+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:38:30.143 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:30.140+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:38:30.155 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:30.152+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:38:33.424 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:38:33.425 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:38:33.426 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:38:33.444 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:38:33.692 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:33.688+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:38:33.710 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:33.708+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:38:34.800 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:38:34.800 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:38:34.801 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365514 2023-10-03T20:38:34.801 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365514 -le 1696367755 ]] 2023-10-03T20:38:34.801 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:38:34.801 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:38:34.802 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:38:41.947 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:41.944+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:38:41.953 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:41.948+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:38:44.820 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:38:44.820 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:38:44.830 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:38:44.842 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:38:46.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:38:46.108 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:38:46.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365526 2023-10-03T20:38:46.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365526 -le 1696367755 ]] 2023-10-03T20:38:46.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:38:46.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:38:46.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:38:53.243 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:38:53.235+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:38:54.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:54.903+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:38:54.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:54.903+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:38:54.918 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:54.911+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:38:54.920 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:54.915+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:38:54.925 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:38:54.919+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:38:56.117 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:38:56.117 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:38:56.117 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:38:56.131 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:38:57.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:38:57.495 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:38:57.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365537 2023-10-03T20:38:57.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365537 -le 1696367755 ]] 2023-10-03T20:38:57.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:38:57.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:38:57.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:39:00.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:00.163+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:39:00.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:00.163+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:39:00.190 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:00.183+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:39:00.198 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:00.191+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:39:00.199 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:00.191+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:39:00.210 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:00.203+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:39:02.514 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:02.507+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:39:02.518 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:02.515+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:39:02.544 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:02.539+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:39:02.566 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:02.551+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:39:07.512 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:39:07.512 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:39:07.512 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:39:07.525 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:39:07.779 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:07.775+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:39:07.779 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:07.775+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:39:08.969 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:39:08.978 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:39:08.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365548 2023-10-03T20:39:08.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365548 -le 1696367755 ]] 2023-10-03T20:39:08.980 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:39:08.980 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:39:08.980 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:39:09.258 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:09.255+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:39:09.259 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:09.255+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:39:09.259 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:09.255+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:39:09.259 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:09.255+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:39:17.133 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:17.127+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:39:17.133 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:17.127+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:39:18.985 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:39:18.991 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:39:18.994 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:39:19.010 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:39:20.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:39:20.367 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:39:20.368 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365560 2023-10-03T20:39:20.369 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365560 -le 1696367755 ]] 2023-10-03T20:39:20.369 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:39:20.369 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:39:20.369 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:39:23.187 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:23.182+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:39:23.187 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:23.182+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:39:23.188 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:23.182+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:39:23.195 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:23.190+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:39:23.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:23.190+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:39:23.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:23.190+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:39:26.907 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:26.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:39:26.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:26.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:39:26.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:26.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:39:26.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:26.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:39:26.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:26.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:39:30.379 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:39:30.380 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:39:30.382 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:39:30.391 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:39:30.581 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:30.574+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:39:30.581 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:30.574+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:39:30.581 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:30.574+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:39:31.723 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:39:31.724 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:39:31.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365571 2023-10-03T20:39:31.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365571 -le 1696367755 ]] 2023-10-03T20:39:31.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:39:31.725 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:39:31.725 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:39:35.994 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:39:35.986+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:39:39.099 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:39.090+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:39:39.117 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:39.102+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:39:39.118 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:39.106+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:39:39.119 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:39.114+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:39:39.119 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:39.114+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:39:41.730 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:39:41.730 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:39:41.733 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:39:41.777 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:39:43.015 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:39:43.015 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:39:43.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365583 2023-10-03T20:39:43.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365583 -le 1696367755 ]] 2023-10-03T20:39:43.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:39:43.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:39:43.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:39:43.888 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:43.882+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:39:43.888 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:43.882+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:39:43.888 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:43.882+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:39:43.888 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:39:43.882+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:39:53.026 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:39:53.027 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:39:53.037 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:39:53.039 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:39:54.435 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:39:54.440 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:39:54.441 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365594 2023-10-03T20:39:54.441 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365594 -le 1696367755 ]] 2023-10-03T20:39:54.442 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:39:54.442 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:39:54.442 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:40:04.460 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:40:04.460 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:40:04.463 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:40:04.478 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:40:06.006 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:40:06.006 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:40:06.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365606 2023-10-03T20:40:06.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365606 -le 1696367755 ]] 2023-10-03T20:40:06.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:40:06.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:40:06.007 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:40:07.352 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.349+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:40:07.352 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.349+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:40:07.352 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.349+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:40:07.354 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.353+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:40:07.356 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.353+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:40:07.356 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.353+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:40:07.356 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:07.353+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:40:16.015 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:40:16.020 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:40:16.020 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:40:16.034 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:40:17.267 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:40:17.272 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:40:17.273 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365617 2023-10-03T20:40:17.273 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365617 -le 1696367755 ]] 2023-10-03T20:40:17.273 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:40:17.273 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:40:17.273 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:40:24.948 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.945+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:40:24.958 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.953+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:40:24.958 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.953+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:40:24.958 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.953+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:40:24.958 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.953+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:40:24.969 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.965+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:40:24.969 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:24.965+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:40:27.274 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:40:27.276 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:40:27.278 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:40:27.283 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:40:28.679 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:40:28.682 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:40:28.683 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365628 2023-10-03T20:40:28.684 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365628 -le 1696367755 ]] 2023-10-03T20:40:28.684 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:40:28.684 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:40:28.684 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:40:30.762 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:30.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:40:30.763 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:30.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:40:30.763 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:30.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:40:30.763 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:30.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:40:32.002 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:31.997+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:40:32.012 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:32.009+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:40:32.012 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:32.009+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:40:32.012 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:32.009+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:40:32.012 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:32.009+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:40:32.012 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:32.009+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:40:33.595 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:33.585+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:40:35.532 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:35.529+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:40:38.685 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:40:38.685 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:40:38.695 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:40:38.696 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:40:39.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:40:39.720 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:40:39.727 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365639 2023-10-03T20:40:39.727 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365639 -le 1696367755 ]] 2023-10-03T20:40:39.727 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:40:39.728 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:40:39.728 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:40:40.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:40.876+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:40:45.820 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:45.816+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:40:45.852 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:45.848+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:40:45.853 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:45.848+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:40:45.853 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:45.848+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:40:49.737 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:40:49.737 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:40:49.737 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:40:49.750 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:40:50.045 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:50.040+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:40:50.045 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:50.040+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:40:50.086 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:50.080+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:40:50.097 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:50.092+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:40:51.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:40:51.013 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:40:51.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365651 2023-10-03T20:40:51.013 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365651 -le 1696367755 ]] 2023-10-03T20:40:51.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:40:51.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:40:51.014 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:40:52.051 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:52.048+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:40:52.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:52.060+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:40:52.070 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:40:52.064+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:40:55.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:55.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:40:55.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:55.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:40:55.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:55.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:40:55.913 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:55.908+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:40:55.913 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:55.908+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:40:55.913 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:40:55.908+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:41:01.026 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:41:01.040 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:41:01.047 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:41:01.056 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:41:02.389 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:41:02.389 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:41:02.397 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365662 2023-10-03T20:41:02.398 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365662 -le 1696367755 ]] 2023-10-03T20:41:02.398 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:41:02.398 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:41:02.398 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:41:03.353 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:03.348+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:41:12.416 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:41:12.422 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:41:12.425 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:41:12.427 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:41:13.932 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:41:13.932 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:41:13.933 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365673 2023-10-03T20:41:13.933 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365673 -le 1696367755 ]] 2023-10-03T20:41:13.933 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:41:13.933 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:41:13.934 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:41:16.834 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:16.828+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:41:16.858 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:16.844+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:41:16.858 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:16.844+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:41:16.858 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:16.844+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:41:19.327 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:19.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:41:19.332 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:19.328+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:41:19.332 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:19.328+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:41:23.935 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:41:23.935 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:41:23.935 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:41:23.935 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:41:24.836 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:24.827+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:41:24.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:24.827+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:41:24.838 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:24.831+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:41:25.127 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:41:25.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:41:25.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365685 2023-10-03T20:41:25.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365685 -le 1696367755 ]] 2023-10-03T20:41:25.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:41:25.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:41:25.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:41:35.133 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:41:35.138 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:41:35.139 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:41:35.142 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:41:36.297 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:41:36.301 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:41:36.310 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365696 2023-10-03T20:41:36.310 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365696 -le 1696367755 ]] 2023-10-03T20:41:36.310 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:41:36.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:41:36.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:41:38.314 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:38.307+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:41:38.314 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:38.307+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:41:38.326 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:38.319+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:41:38.326 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:38.319+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:41:38.335 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:38.331+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:41:45.480 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:45.475+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:41:45.496 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:45.491+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:41:45.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:45.495+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:41:45.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:45.495+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:41:46.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:41:46.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:41:46.327 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:41:46.330 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:41:47.581 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:41:47.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:41:47.589 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365707 2023-10-03T20:41:47.589 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365707 -le 1696367755 ]] 2023-10-03T20:41:47.589 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:41:47.590 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:41:47.590 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:41:48.377 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:48.371+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:41:48.398 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:48.391+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:41:50.659 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:50.651+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:41:50.663 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:50.655+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:41:50.669 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:50.663+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:41:52.090 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:52.079+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:41:52.101 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:52.083+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:41:52.101 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:52.083+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:41:52.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:52.107+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:41:52.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:52.107+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:41:52.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:41:52.107+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:41:52.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:52.899+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:41:52.915 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:52.907+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:41:53.744 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:41:53.735+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:41:57.608 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:41:57.613 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:41:57.615 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:41:57.637 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:41:59.278 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:41:59.284 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:41:59.285 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365719 2023-10-03T20:41:59.286 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365719 -le 1696367755 ]] 2023-10-03T20:41:59.286 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:41:59.286 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:41:59.286 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:42:01.623 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:01.615+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:42:01.623 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:01.615+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:42:01.623 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:01.615+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:42:01.632 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:01.627+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:42:01.632 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:01.627+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:42:01.632 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:01.627+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:42:06.366 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.359+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:42:06.366 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.359+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:42:06.367 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.359+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:42:06.371 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.367+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:42:06.371 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.367+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:42:06.388 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.383+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:42:06.388 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:06.383+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:42:09.293 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:42:09.293 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:42:09.293 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:42:09.294 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:42:10.312 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:10.306+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:42:10.313 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:10.306+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:42:10.609 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:42:10.616 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:42:10.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365730 2023-10-03T20:42:10.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365730 -le 1696367755 ]] 2023-10-03T20:42:10.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:42:10.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:42:10.618 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:42:20.620 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:42:20.620 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:42:20.646 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:42:20.658 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:42:21.928 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:42:21.930 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:42:21.931 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365741 2023-10-03T20:42:21.931 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365741 -le 1696367755 ]] 2023-10-03T20:42:21.931 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:42:21.931 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:42:21.932 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:42:23.283 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:23.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:42:23.295 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:23.290+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:42:26.653 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:42:26.653 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:42:26.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:42:26.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:42:26.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:42:26.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:42:26.673 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:42:26.666+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:42:31.940 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:42:31.940 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:42:31.940 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:42:31.941 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:42:33.265 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:33.254+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:42:33.265 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:33.254+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:42:33.408 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:42:33.411 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:42:33.412 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365753 2023-10-03T20:42:33.412 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365753 -le 1696367755 ]] 2023-10-03T20:42:33.412 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:42:33.412 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:42:33.412 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:42:43.426 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:42:43.427 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:42:43.428 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:42:43.432 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:42:44.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:42:44.976 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:42:44.977 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365764 2023-10-03T20:42:44.977 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365764 -le 1696367755 ]] 2023-10-03T20:42:44.978 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:42:44.978 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:42:44.978 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:42:50.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:50.494+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:42:50.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:50.494+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:42:50.507 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:50.506+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:42:50.514 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:50.510+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:42:50.514 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:50.510+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:42:51.463 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:51.458+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:42:54.249 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:54.246+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:42:54.267 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:42:54.262+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:42:54.979 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:42:54.980 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:42:54.988 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:42:54.988 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:42:56.293 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:42:56.293 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:42:56.294 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365776 2023-10-03T20:42:56.294 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365776 -le 1696367755 ]] 2023-10-03T20:42:56.294 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:42:56.294 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:42:56.294 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:43:05.155 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:05.153+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:43:05.185 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:05.181+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:43:05.185 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:05.181+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:05.185 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:05.181+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:06.306 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:43:06.311 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:43:06.314 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:43:06.316 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:43:07.693 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:43:07.694 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:43:07.694 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365787 2023-10-03T20:43:07.694 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365787 -le 1696367755 ]] 2023-10-03T20:43:07.694 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:43:07.694 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:43:07.695 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:43:09.586 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:09.577+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:43:11.625 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:11.621+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:43:11.644 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:11.641+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:43:12.589 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:12.581+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:12.595 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:12.589+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:13.281 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:13.277+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:43:13.303 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:13.301+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:43:17.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.645+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:43:17.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.645+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:43:17.652 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.645+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:43:17.662 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.657+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:43:17.662 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.657+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:43:17.662 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.657+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:43:17.662 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.657+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:43:17.663 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.657+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:43:17.663 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:17.657+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:43:17.705 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:43:17.710 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:43:17.713 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:43:17.721 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:43:18.834 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:43:18.835 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:43:18.835 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365798 2023-10-03T20:43:18.835 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365798 -le 1696367755 ]] 2023-10-03T20:43:18.836 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:43:18.836 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:43:18.836 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:43:28.841 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:43:28.842 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:28.833+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:43:28.849 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:43:28.851 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:43:28.860 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:43:29.204 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:29.201+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:43:29.204 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:29.201+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:43:29.205 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:29.201+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:43:29.230 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:29.225+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:29.230 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:29.225+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:43:29.230 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:29.225+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:30.183 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:43:30.183 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:43:30.184 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365810 2023-10-03T20:43:30.184 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365810 -le 1696367755 ]] 2023-10-03T20:43:30.184 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:43:30.185 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:43:30.185 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:43:31.449 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:31.445+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:31.459 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:31.453+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:37.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:37.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:43:37.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:37.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:43:37.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:37.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:37.332 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:37.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:39.836 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:39.833+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:39.851 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:39.845+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:40.197 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:43:40.198 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:43:40.205 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:43:40.214 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:43:41.571 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:43:41.572 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:43:41.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365821 2023-10-03T20:43:41.574 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365821 -le 1696367755 ]] 2023-10-03T20:43:41.574 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:43:41.574 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:43:41.574 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:43:45.160 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:45.157+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:43:45.167 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:45.161+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:43:45.167 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:45.161+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:43:45.167 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:45.161+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:43:45.168 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:45.161+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:43:45.168 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:43:45.161+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:43:46.557 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:46.553+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:43:46.558 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:46.553+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:43:46.570 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:46.565+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:43:46.577 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:46.573+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:43:46.577 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:43:46.573+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:43:51.581 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:43:51.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:43:51.583 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:43:51.588 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:43:52.951 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:43:52.954 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:43:52.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365832 2023-10-03T20:43:52.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365832 -le 1696367755 ]] 2023-10-03T20:43:52.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:43:52.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:43:52.956 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:44:01.060 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.052+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:44:01.060 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.052+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:44:01.060 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.052+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:44:01.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:44:01.067 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:44:01.067 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:44:01.067 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:01.060+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:44:02.961 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:44:02.961 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:44:02.962 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:44:02.962 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:44:04.199 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:44:04.199 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:44:04.200 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365844 2023-10-03T20:44:04.201 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365844 -le 1696367755 ]] 2023-10-03T20:44:04.201 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:44:04.201 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:44:04.201 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:44:05.125 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:05.112+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:44:12.137 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:12.132+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:44:12.137 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:12.132+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:44:12.159 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:12.152+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:44:12.159 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:12.152+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:44:12.159 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:12.152+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:44:14.206 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:44:14.213 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:44:14.213 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:44:14.219 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:44:15.440 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:44:15.448 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:44:15.448 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365855 2023-10-03T20:44:15.448 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365855 -le 1696367755 ]] 2023-10-03T20:44:15.449 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:44:15.449 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:44:15.449 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:44:15.766 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:15.752+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:44:16.620 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:16.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:44:18.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.472+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:44:18.491 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.484+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:18.492 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.484+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:44:18.492 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.484+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:44:18.492 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.484+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:18.492 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.484+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:44:18.492 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:18.484+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:44:22.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:44:22.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:22.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:44:22.331 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:44:22.338 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.332+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:22.351 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.344+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:44:22.351 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:22.344+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:44:25.468 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:44:25.474 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:44:25.485 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:44:25.491 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:44:26.392 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:26.384+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:26.408 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:26.404+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:44:26.782 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:44:26.782 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:44:26.782 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365866 2023-10-03T20:44:26.783 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365866 -le 1696367755 ]] 2023-10-03T20:44:26.783 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:44:26.783 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:44:26.783 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:44:28.437 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:28.432+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:28.438 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:28.432+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:44:28.438 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:28.432+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:44:28.479 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:28.476+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:28.480 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:28.476+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:44:31.901 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:31.896+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:44:31.901 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:31.896+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:44:31.917 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:31.912+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:44:31.918 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:31.912+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:44:33.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:33.992+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:44:33.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:33.992+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:44:34.007 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:34.004+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:44:34.014 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:34.008+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:44:34.014 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:34.008+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:44:34.023 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:34.020+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:44:35.208 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:35.204+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:44:35.210 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:35.204+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:44:35.214 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:35.208+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:44:35.791 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:35.784+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:35.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:35.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:44:36.791 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:44:36.791 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:44:36.793 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:44:36.793 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:44:38.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:44:38.314 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:44:38.315 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365878 2023-10-03T20:44:38.315 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365878 -le 1696367755 ]] 2023-10-03T20:44:38.315 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:44:38.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:44:38.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:44:40.516 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:40.508+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:40.516 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:40.508+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:44:40.525 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:40.520+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:41.503 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:41.488+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:41.504 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:41.488+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:44:41.504 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:41.488+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:44:41.511 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:41.504+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:41.511 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:41.504+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:44:46.373 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:46.367+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:44:46.373 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:46.367+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:44:46.373 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:46.367+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:44:46.416 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:46.411+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:44:46.416 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:46.411+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:44:46.442 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:44:46.435+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:44:48.317 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:44:48.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:44:48.321 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:44:48.328 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:44:49.569 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:44:49.575 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:44:49.575 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365889 2023-10-03T20:44:49.575 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365889 -le 1696367755 ]] 2023-10-03T20:44:49.575 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:44:49.575 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:44:49.575 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:44:52.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.247+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:44:52.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.247+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:44:52.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.247+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:44:52.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.263+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:44:52.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.263+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:44:52.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.263+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:44:52.318 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.303+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:44:52.318 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.303+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:44:52.318 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.303+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:44:52.318 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.303+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:44:52.318 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.303+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:44:52.319 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:44:52.303+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:44:59.578 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:44:59.578 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:44:59.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:44:59.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:45:00.854 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:45:00.855 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:45:00.855 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365900 2023-10-03T20:45:00.855 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365900 -le 1696367755 ]] 2023-10-03T20:45:00.855 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:45:00.856 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:45:00.856 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:45:06.201 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:06.195+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:45:06.201 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:06.195+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:45:06.214 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:06.207+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:45:06.215 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:06.207+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:45:06.215 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:06.207+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:45:06.899 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:06.891+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:45:06.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:06.891+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:45:06.909 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:06.903+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:45:06.920 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:06.915+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:45:06.928 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:06.915+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:45:10.860 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:45:10.861 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:45:10.864 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:45:10.867 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:45:12.278 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:45:12.278 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:45:12.279 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365912 2023-10-03T20:45:12.279 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365912 -le 1696367755 ]] 2023-10-03T20:45:12.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:45:12.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:45:12.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:45:22.296 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:45:22.296 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:45:22.305 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:45:22.313 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:45:23.364 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:45:23.365 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:45:23.366 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365923 2023-10-03T20:45:23.366 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365923 -le 1696367755 ]] 2023-10-03T20:45:23.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:45:23.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:45:23.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:45:29.421 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:29.415+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:45:29.433 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:29.427+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:45:31.727 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:31.723+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:45:31.743 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:31.735+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:45:31.743 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:31.735+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:45:31.743 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:31.735+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:45:33.369 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:45:33.382 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:45:33.385 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:45:33.385 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:45:34.676 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:45:34.687 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:45:34.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365934 2023-10-03T20:45:34.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365934 -le 1696367755 ]] 2023-10-03T20:45:34.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:45:34.688 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:45:34.689 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:45:38.072 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:38.063+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:45:38.072 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:38.063+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:45:38.072 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:38.063+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:45:39.302 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:39.295+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:45:39.353 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:39.347+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:45:39.353 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:39.347+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:45:39.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:45:39.363+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:45:44.693 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:45:44.693 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:45:44.693 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:45:44.700 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:45:44.770 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.746+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:45:44.794 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.782+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:45:44.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.798+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:45:44.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.798+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:45:44.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.798+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:45:44.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.798+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:45:44.804 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:44.798+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:45:45.942 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:45:45.942 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:45:45.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365945 2023-10-03T20:45:45.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365945 -le 1696367755 ]] 2023-10-03T20:45:45.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:45:45.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:45:45.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:45:46.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:46.270+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:45:46.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:45:46.270+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:45:55.947 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:45:55.952 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:45:55.953 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:45:55.961 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:45:57.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:45:57.179 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:45:57.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365957 2023-10-03T20:45:57.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365957 -le 1696367755 ]] 2023-10-03T20:45:57.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:45:57.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:45:57.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:46:02.608 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:02.598+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:02.612 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:02.606+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:02.624 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:02.614+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:04.682 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.674+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:46:04.693 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.686+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:04.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:46:04.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:46:04.707 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:04.707 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:46:04.707 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.698+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:04.809 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.802+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:04.816 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.810+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:46:04.816 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.810+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:04.817 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:04.810+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:05.266 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:05.250+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:05.267 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:05.262+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:05.284 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:05.278+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:07.183 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:46:07.189 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:46:07.190 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:46:07.190 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:46:08.315 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:46:08.315 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:46:08.315 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365968 2023-10-03T20:46:08.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365968 -le 1696367755 ]] 2023-10-03T20:46:08.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:46:08.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:46:08.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:46:11.361 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:11.354+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:46:11.372 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:11.366+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:46:18.317 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:46:18.345 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:46:18.350 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:46:18.359 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:46:18.640 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.634+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:46:18.640 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.634+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:46:18.641 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.634+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:46:18.670 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.658+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:46:18.671 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.658+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:46:18.671 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.662+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:46:18.690 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.678+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:46:18.690 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.678+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:46:18.690 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.678+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:46:18.690 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.678+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:46:18.691 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.682+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:46:18.691 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.682+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:46:18.691 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.682+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:46:18.691 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:18.682+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:46:19.756 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:46:19.756 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:46:19.758 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365979 2023-10-03T20:46:19.758 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365979 -le 1696367755 ]] 2023-10-03T20:46:19.759 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:46:19.759 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:46:19.759 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:46:23.284 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:23.274+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:46:23.284 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:23.274+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:46:23.299 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:23.290+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:46:23.310 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:23.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:46:23.311 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:23.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:46:24.236 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:24.230+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:46:24.244 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:24.238+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:46:24.244 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:24.238+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:46:24.245 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:24.238+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:46:27.283 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:27.278+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:27.284 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:27.278+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:46:27.284 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:27.278+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:46:27.300 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:27.298+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:27.300 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:27.298+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:46:29.761 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:46:29.761 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:46:29.761 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:46:29.787 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:46:31.166 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:46:31.166 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:46:31.166 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696365991 2023-10-03T20:46:31.167 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696365991 -le 1696367755 ]] 2023-10-03T20:46:31.167 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:46:31.167 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:46:31.167 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:46:33.962 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:46:33.963 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:33.963 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:46:33.963 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:46:33.963 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:33.964 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:46:33.964 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:33.958+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:34.181 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:34.178+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:46:34.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:34.194+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:34.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:34.194+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:46:34.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:34.194+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:46:34.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:34.194+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:34.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:34.194+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:41.168 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:46:41.172 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:46:41.174 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:46:41.174 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:46:42.427 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:46:42.434 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:46:42.435 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366002 2023-10-03T20:46:42.435 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366002 -le 1696367755 ]] 2023-10-03T20:46:42.435 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:46:42.436 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:46:42.436 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:46:52.437 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:46:52.437 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:46:52.437 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:46:52.446 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:46:53.880 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:46:53.880 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:46:53.881 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366013 2023-10-03T20:46:53.881 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366013 -le 1696367755 ]] 2023-10-03T20:46:53.881 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:46:53.882 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:46:53.882 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:46:57.894 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:57.889+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:46:57.899 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:57.893+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:46:59.094 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.089+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:46:59.106 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.097+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:46:59.106 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.097+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:46:59.107 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.097+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:46:59.107 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.101+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:46:59.127 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.125+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:46:59.127 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:46:59.125+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:46:59.263 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:59.253+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:46:59.263 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:46:59.253+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:47:00.894 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:00.889+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:01.936 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:01.933+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:47:03.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.629+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:47:03.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.629+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:47:03.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.629+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:47:03.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.629+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:47:03.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.629+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:47:03.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.629+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:47:03.806 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.801+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:47:03.828 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.817+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:47:03.828 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.817+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:47:03.828 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.817+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:47:03.833 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.829+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:47:03.833 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.829+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:47:03.840 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:03.837+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:47:03.884 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:47:03.887 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:47:03.889 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:47:03.892 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:47:04.176 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.173+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:47:04.176 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.173+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:47:04.176 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.173+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:47:04.179 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.173+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:47:04.179 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.173+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:47:04.197 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.193+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:04.200 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.197+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:47:04.215 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.205+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:47:04.215 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:04.205+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:47:05.107 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:47:05.108 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:47:05.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366025 2023-10-03T20:47:05.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366025 -le 1696367755 ]] 2023-10-03T20:47:05.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:47:05.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:47:05.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:47:05.552 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:05.545+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:05.552 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:05.545+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:47:15.148 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:47:15.149 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:47:15.150 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:47:15.160 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:47:16.355 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:47:16.355 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:47:16.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366036 2023-10-03T20:47:16.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366036 -le 1696367755 ]] 2023-10-03T20:47:16.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:47:16.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:47:16.356 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:47:20.062 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.057+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:47:20.062 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.057+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:47:20.077 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.073+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:20.086 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.081+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:47:20.086 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.081+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:47:20.086 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.081+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:47:20.090 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.085+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:47:20.097 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:20.093+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:47:26.371 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:47:26.373 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:47:26.378 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:47:26.379 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:47:27.834 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:47:27.835 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:47:27.836 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366047 2023-10-03T20:47:27.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366047 -le 1696367755 ]] 2023-10-03T20:47:27.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:47:27.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:47:27.837 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:47:29.032 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:29.029+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:35.012 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.004+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:47:35.012 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.004+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:47:35.012 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.004+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:47:35.028 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:47:35.029 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:35.016+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:47:37.372 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:37.360+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:47:37.373 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:37.360+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:47:37.382 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:37.376+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:37.389 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:37.380+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:47:37.464 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:37.444+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:47:37.464 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:37.444+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:47:37.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:37.460+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:47:37.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:37.460+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:47:37.854 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:47:37.854 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:47:37.855 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:47:37.868 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:47:39.036 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:47:39.038 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:47:39.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366059 2023-10-03T20:47:39.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366059 -le 1696367755 ]] 2023-10-03T20:47:39.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:47:39.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:47:39.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:47:39.967 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:39.960+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:47:39.968 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:39.960+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:47:39.968 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:39.964+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:47:39.978 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:39.972+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:47:39.978 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:39.972+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:47:43.573 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:43.568+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:47:49.057 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:47:49.059 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:47:49.061 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:47:49.061 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:47:50.303 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:47:50.303 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:47:50.303 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366070 2023-10-03T20:47:50.303 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366070 -le 1696367755 ]] 2023-10-03T20:47:50.303 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:47:50.303 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:47:50.304 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:47:53.898 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:53.892+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:47:53.898 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:53.892+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:47:53.899 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:53.892+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:47:53.912 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:53.904+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:47:53.913 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:47:53.904+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:47:58.846 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:58.840+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:47:58.884 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:47:58.880+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:48:00.306 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:48:00.307 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:48:00.313 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:48:00.343 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:48:01.631 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:48:01.631 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:48:01.632 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366081 2023-10-03T20:48:01.632 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366081 -le 1696367755 ]] 2023-10-03T20:48:01.632 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:48:01.632 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:48:01.632 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:48:04.933 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:04.928+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:11.641 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:48:11.641 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:48:11.643 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:48:11.695 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:48:11.919 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:11.912+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:48:11.919 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:11.912+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:48:11.945 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:11.940+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:11.950 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:11.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:48:11.950 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:11.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:48:12.474 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:48:12.474 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:48:12.479 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366092 2023-10-03T20:48:12.479 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366092 -le 1696367755 ]] 2023-10-03T20:48:12.479 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:48:12.479 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:48:12.480 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:48:14.216 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:14.200+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:48:14.216 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:14.200+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:48:14.216 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:14.200+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:48:14.216 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:14.200+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:48:14.216 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:14.200+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:48:17.504 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:17.500+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:48:17.504 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:17.500+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:48:17.517 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:17.512+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:17.522 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:17.516+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:48:17.522 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:17.516+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:48:19.813 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:19.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:48:19.814 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:19.808+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:48:19.825 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:19.820+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:48:19.825 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:19.820+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:48:22.482 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:48:22.482 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:48:22.482 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:48:22.485 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:48:22.903 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.896+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:48:22.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.904+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:22.925 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.916+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:48:22.934 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.928+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:48:22.961 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.956+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:22.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.968+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:48:22.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:22.968+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:48:23.572 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:48:23.579 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:48:23.579 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366103 2023-10-03T20:48:23.579 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366103 -le 1696367755 ]] 2023-10-03T20:48:23.579 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:48:23.579 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:48:23.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:48:24.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:24.876+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:48:24.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:24.884+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:48:30.152 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:30.143+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:48:30.152 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:30.143+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:48:30.152 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:30.143+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:31.209 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:31.203+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:48:31.217 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:31.211+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:48:31.217 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:31.211+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:48:31.217 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:31.211+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:48:31.217 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:31.211+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:48:33.590 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:48:33.590 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:48:33.593 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:48:33.603 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:48:34.848 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:48:34.848 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:48:34.856 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366114 2023-10-03T20:48:34.856 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366114 -le 1696367755 ]] 2023-10-03T20:48:34.857 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:48:34.857 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:48:34.857 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:48:44.866 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:48:44.869 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:48:44.871 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:48:44.909 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:48:46.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:48:46.177 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:48:46.179 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366126 2023-10-03T20:48:46.179 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366126 -le 1696367755 ]] 2023-10-03T20:48:46.179 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:48:46.179 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:48:46.179 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:48:56.181 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:48:56.181 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:48:56.198 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:48:56.216 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:48:57.344 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:48:57.349 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:48:57.350 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366137 2023-10-03T20:48:57.350 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366137 -le 1696367755 ]] 2023-10-03T20:48:57.350 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:48:57.351 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:48:57.351 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:48:57.647 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:57.639+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:48:57.664 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:48:57.659+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:48:58.484 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:58.471+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:48:58.494 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:58.487+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:48:58.499 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:58.495+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:48:58.500 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:48:58.495+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:49:01.215 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:01.211+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:49:01.219 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:01.211+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:49:01.219 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:01.211+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:01.225 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:01.215+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:49:03.566 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:03.559+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:03.920 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:03.915+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:49:03.924 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:03.919+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:49:06.842 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.835+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:49:06.842 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.835+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T20:49:06.842 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.835+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:49:06.845 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.839+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:49:06.845 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.839+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:49:06.846 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.839+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:06.846 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.839+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:49:06.846 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.839+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:49:06.846 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:06.839+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:49:07.354 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:49:07.354 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:49:07.354 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:49:07.359 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:49:08.722 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:49:08.723 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:49:08.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366148 2023-10-03T20:49:08.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366148 -le 1696367755 ]] 2023-10-03T20:49:08.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:49:08.725 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:49:08.725 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:49:14.531 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:14.519+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:49:14.531 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:14.519+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:49:14.531 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:14.519+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:49:14.744 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:14.735+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:49:14.745 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:14.739+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:49:14.745 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:14.739+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:49:18.729 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:49:18.730 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:49:18.739 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:49:18.751 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:49:19.985 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:49:19.985 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:49:19.986 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366159 2023-10-03T20:49:19.986 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366159 -le 1696367755 ]] 2023-10-03T20:49:19.986 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:49:19.986 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:49:19.986 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:49:25.285 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:25.270+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:49:25.306 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:25.294+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:49:29.988 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:49:29.996 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:49:30.006 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:49:30.013 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:49:31.232 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:49:31.232 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:49:31.233 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366171 2023-10-03T20:49:31.233 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366171 -le 1696367755 ]] 2023-10-03T20:49:31.233 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:49:31.233 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:49:31.233 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:49:32.339 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:32.326+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:49:32.339 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:32.326+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:49:32.340 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:32.330+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:32.369 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:32.362+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:49:32.369 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:32.362+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:49:32.369 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:32.362+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:49:33.381 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:33.374+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:34.716 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:34.710+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:49:34.716 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:34.710+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:49:34.729 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:34.722+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:49:34.734 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:34.726+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:49:34.765 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:34.758+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:49:36.923 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.914+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:49:36.930 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:49:36.931 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:49:36.931 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:49:36.931 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:49:36.931 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:49:36.931 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:49:36.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:49:41.242 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:49:41.242 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:49:41.264 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:49:41.272 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:49:42.666 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:49:42.673 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:49:42.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366182 2023-10-03T20:49:42.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366182 -le 1696367755 ]] 2023-10-03T20:49:42.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:49:42.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:49:42.674 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:49:47.905 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:47.894+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:52.677 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:49:52.677 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:49:52.683 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:49:52.683 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:49:53.896 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:49:53.897 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:49:53.898 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366193 2023-10-03T20:49:53.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366193 -le 1696367755 ]] 2023-10-03T20:49:53.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:49:53.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:49:53.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:49:56.227 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:56.218+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:49:56.227 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:56.218+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:49:56.236 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:56.230+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:49:58.416 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:58.410+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:49:58.439 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:49:58.434+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:50:03.910 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:50:03.911 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:50:03.911 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:50:03.936 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:50:05.167 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:50:05.167 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:50:05.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366205 2023-10-03T20:50:05.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366205 -le 1696367755 ]] 2023-10-03T20:50:05.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:50:05.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:50:05.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:50:15.179 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:50:15.196 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:50:15.196 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:50:15.200 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:50:16.474 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:50:16.474 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:50:16.475 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366216 2023-10-03T20:50:16.475 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366216 -le 1696367755 ]] 2023-10-03T20:50:16.475 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:50:16.475 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:50:16.475 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:50:18.553 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:50:18.558 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.554+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:50:18.558 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.554+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:50:18.558 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.554+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:50:18.560 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.558+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:50:18.560 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.558+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:50:18.560 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:18.558+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:50:20.656 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:20.654+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:50:20.657 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:20.654+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:50:20.657 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:20.654+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:50:20.657 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:20.654+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:50:20.657 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:20.654+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:50:24.350 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:24.346+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:50:24.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:24.366+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:50:24.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:24.366+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:50:24.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:24.366+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:50:24.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:24.366+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:50:24.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:24.366+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:50:26.488 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:50:26.488 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:50:26.495 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:50:26.495 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:50:27.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:50:27.723 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:50:27.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366227 2023-10-03T20:50:27.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366227 -le 1696367755 ]] 2023-10-03T20:50:27.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:50:27.724 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:50:27.725 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:50:32.354 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:32.349+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:50:37.731 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:50:37.731 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:50:37.731 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:50:37.753 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:50:38.954 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:50:38.954 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:50:38.954 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366238 2023-10-03T20:50:38.954 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366238 -le 1696367755 ]] 2023-10-03T20:50:38.954 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:50:38.955 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:50:38.955 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:50:40.776 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:40.769+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:50:40.783 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:40.769+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:50:40.783 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:40.769+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:50:40.790 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:40.785+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:50:40.790 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:40.785+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:50:41.899 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:41.893+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:50:41.900 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:41.893+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:50:41.917 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:41.913+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:50:41.929 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:41.925+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:50:41.930 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:41.925+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:50:41.930 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:41.925+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:50:47.379 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:50:47.377+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:50:47.797 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.793+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:50:47.797 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.793+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:50:47.806 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.801+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:50:47.832 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.829+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:50:47.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.833+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:50:47.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.833+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:50:47.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.833+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:50:47.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:47.833+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:50:48.975 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:50:48.976 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:50:48.977 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:50:48.988 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:50:50.228 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:50:50.229 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:50:50.230 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366250 2023-10-03T20:50:50.230 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366250 -le 1696367755 ]] 2023-10-03T20:50:50.231 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:50:50.231 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:50:50.231 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:50:51.873 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:51.861+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:50:51.876 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:51.861+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:50:51.878 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:50:51.873+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:51:00.248 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:51:00.248 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:51:00.249 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:51:00.263 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:51:00.918 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:00.913+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:51:00.929 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:00.925+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:51:01.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:51:01.467 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:51:01.469 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366261 2023-10-03T20:51:01.469 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366261 -le 1696367755 ]] 2023-10-03T20:51:01.469 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:51:01.469 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:51:01.470 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:51:08.519 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:08.513+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:51:08.527 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:08.525+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:51:08.527 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:08.525+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:51:11.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:51:11.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:51:11.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:51:11.485 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:51:12.859 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:51:12.863 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:51:12.865 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366272 2023-10-03T20:51:12.865 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366272 -le 1696367755 ]] 2023-10-03T20:51:12.865 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:51:12.865 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:51:12.865 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:51:13.539 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:13.533+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:51:13.545 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:13.541+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:51:13.545 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:13.541+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:51:13.551 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:13.549+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:51:13.560 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:13.553+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:51:13.562 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:13.553+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:51:13.785 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:13.781+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:51:13.785 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:13.781+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:51:13.801 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:13.793+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:51:13.802 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:13.793+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:51:18.234 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.229+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:51:18.261 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.257+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:51:18.261 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.257+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:51:18.267 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.261+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:51:18.275 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.269+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:51:18.286 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.281+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:51:18.286 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:18.281+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:51:21.014 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:21.005+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:51:22.237 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:22.229+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:51:22.237 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:22.229+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:51:22.250 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:22.245+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:51:22.263 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:22.261+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:51:22.263 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:22.261+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:51:22.265 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:22.261+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:51:22.874 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:51:22.879 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:51:22.880 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:51:22.880 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:51:24.371 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:51:24.371 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:51:24.373 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366284 2023-10-03T20:51:24.373 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366284 -le 1696367755 ]] 2023-10-03T20:51:24.373 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:51:24.373 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:51:24.373 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:51:34.376 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:51:34.376 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:51:34.377 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:51:34.405 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:51:35.725 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:51:35.725 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:51:35.726 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366295 2023-10-03T20:51:35.726 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366295 -le 1696367755 ]] 2023-10-03T20:51:35.726 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:51:35.726 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:51:35.726 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:51:39.732 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:39.728+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:51:45.740 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:51:45.740 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:51:45.744 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:51:45.773 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:51:47.180 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:51:47.180 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:51:47.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366307 2023-10-03T20:51:47.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366307 -le 1696367755 ]] 2023-10-03T20:51:47.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:51:47.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:51:47.181 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:51:57.188 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:51:57.189 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:51:57.195 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:51:57.205 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:51:57.479 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:51:57.472+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:51:58.572 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:51:58.572 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:51:58.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366318 2023-10-03T20:51:58.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366318 -le 1696367755 ]] 2023-10-03T20:51:58.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:51:58.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:51:58.573 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:51:59.362 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:59.352+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:51:59.416 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:59.404+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:51:59.417 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:59.408+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:51:59.433 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:51:59.424+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:52:00.625 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:00.620+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:52:06.921 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:06.916+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:52:08.574 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:52:08.575 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:52:08.580 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:52:08.585 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:52:09.880 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:52:09.881 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:52:09.882 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366329 2023-10-03T20:52:09.882 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366329 -le 1696367755 ]] 2023-10-03T20:52:09.883 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:52:09.883 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:52:09.883 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:52:12.311 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:12.304+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:52:12.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:12.324+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:52:12.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:12.324+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:52:12.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:12.324+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:52:12.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:12.324+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:52:12.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:12.324+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:52:12.545 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:12.540+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:52:12.575 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:12.568+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:52:12.575 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:12.568+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:52:18.581 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:18.576+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:52:18.582 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:18.576+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:52:18.607 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:18.600+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:52:19.885 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:52:19.885 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:52:19.887 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:52:19.891 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:52:21.132 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:52:21.132 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:52:21.133 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366341 2023-10-03T20:52:21.133 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366341 -le 1696367755 ]] 2023-10-03T20:52:21.133 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:52:21.133 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:52:21.133 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:52:30.757 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.743+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:52:30.757 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.743+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:52:30.757 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.743+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:52:30.757 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.743+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:52:30.765 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.759+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:52:30.765 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.759+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:52:30.765 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.759+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:52:30.765 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.759+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:52:30.766 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:30.759+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:52:31.134 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:52:31.134 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:52:31.135 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:52:31.195 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:52:32.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:52:32.366 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:52:32.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366352 2023-10-03T20:52:32.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366352 -le 1696367755 ]] 2023-10-03T20:52:32.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:52:32.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:52:32.367 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:52:42.370 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:52:42.370 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:52:42.370 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:52:42.370 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:52:43.778 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:52:43.778 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:52:43.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366363 2023-10-03T20:52:43.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366363 -le 1696367755 ]] 2023-10-03T20:52:43.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:52:43.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:52:43.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:52:43.807 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:43.791+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:52:43.808 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:43.795+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:52:43.808 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:43.795+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:52:43.808 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:43.795+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:52:43.814 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:43.807+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:52:43.825 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:43.815+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:52:53.788 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:52:53.795 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:52:53.798 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:52:53.801 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:52:53.930 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:53.915+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:52:53.930 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:53.915+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:52:53.937 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:53.927+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:52:53.941 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:53.935+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:52:53.941 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:53.935+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:52:53.941 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:52:53.935+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:52:55.091 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:52:55.091 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:52:55.092 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366375 2023-10-03T20:52:55.092 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366375 -le 1696367755 ]] 2023-10-03T20:52:55.093 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:52:55.093 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:52:55.093 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:52:59.502 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:59.495+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:52:59.508 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:52:59.503+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:01.689 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.683+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:53:01.698 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.691+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:53:01.698 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.691+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:53:01.699 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.691+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:53:01.702 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.695+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:01.709 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.703+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:53:01.710 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:01.703+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:05.096 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:53:05.096 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:53:05.098 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:53:05.111 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:53:05.340 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:05.335+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:53:05.340 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:05.335+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:53:05.340 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:05.335+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:53:05.377 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:05.371+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:05.377 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:05.371+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:53:05.377 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:05.371+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:06.329 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:53:06.329 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:53:06.330 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366386 2023-10-03T20:53:06.330 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366386 -le 1696367755 ]] 2023-10-03T20:53:06.330 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:53:06.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:53:06.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:53:08.408 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:08.403+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:53:08.412 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:08.407+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:53:14.994 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:14.987+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:15.040 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:15.035+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:53:15.040 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:15.035+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:16.332 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:53:16.333 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:53:16.336 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:53:16.337 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:53:17.379 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:17.371+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:53:17.399 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:17.391+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:53:17.613 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:53:17.618 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:53:17.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366397 2023-10-03T20:53:17.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366397 -le 1696367755 ]] 2023-10-03T20:53:17.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:53:17.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:53:17.619 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:53:27.631 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:53:27.632 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:53:27.638 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:53:27.646 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:53:29.308 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:53:29.309 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:53:29.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366409 2023-10-03T20:53:29.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366409 -le 1696367755 ]] 2023-10-03T20:53:29.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:53:29.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:53:29.311 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:53:30.487 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:30.482+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:53:30.487 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:30.482+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:53:30.494 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:30.486+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:53:36.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:36.158+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T20:53:36.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:36.158+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:53:36.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:36.158+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:53:36.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:36.158+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:53:36.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:36.158+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:53:39.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:53:39.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:53:39.323 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:53:39.328 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:53:39.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:39.990+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:53:39.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:39.990+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:53:40.017 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:40.006+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:53:40.032 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:40.026+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:53:40.032 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:40.026+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:53:40.032 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:40.026+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:53:40.772 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:53:40.772 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:53:40.772 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366420 2023-10-03T20:53:40.772 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366420 -le 1696367755 ]] 2023-10-03T20:53:40.772 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:53:40.772 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:53:40.773 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:53:41.450 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:41.442+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:41.468 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:41.458+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:43.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:43.378+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:53:43.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:43.378+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:53:43.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:43.378+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:53:43.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:43.378+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:53:43.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:43.378+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:53:43.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:43.378+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:53:46.060 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:46.046+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:53:46.060 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:46.050+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:53:46.061 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:46.054+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:53:46.072 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:46.066+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:46.074 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:46.066+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:53:46.075 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:46.066+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:50.774 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:53:50.774 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:53:50.776 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:53:50.778 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:53:50.820 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:53:50.814+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:53:51.899 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:51.886+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:53:51.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:51.886+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:53:51.900 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:51.894+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:51.920 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:51.914+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:53:51.923 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:51.914+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:53:52.379 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:53:52.383 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:53:52.385 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366432 2023-10-03T20:53:52.385 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366432 -le 1696367755 ]] 2023-10-03T20:53:52.385 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:53:52.385 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:53:52.385 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:53:53.652 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:53.646+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:53:53.669 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:53.662+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:53:53.669 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:53:53.662+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:02.403 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:54:02.413 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:54:02.417 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:54:02.432 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:54:03.893 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:54:03.894 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:54:03.894 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366443 2023-10-03T20:54:03.895 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366443 -le 1696367755 ]] 2023-10-03T20:54:03.895 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:54:03.895 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:54:03.895 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:54:06.363 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:06.350+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:54:06.369 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:06.362+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:06.378 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:06.370+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:54:08.667 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:08.658+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:10.625 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:10.618+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:54:10.626 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:10.618+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:54:10.626 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:10.618+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:54:10.641 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:10.634+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:54:10.649 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:10.642+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:54:10.651 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:10.642+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:11.742 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:11.734+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:11.757 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:11.750+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:54:13.860 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:13.854+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:54:13.861 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:13.858+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:54:13.869 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:13.866+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:54:13.869 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:13.866+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:13.896 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:54:13.906 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:54:13.907 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:54:13.926 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:54:14.835 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:14.830+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:14.843 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:14.838+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:54:15.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:54:15.401 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:54:15.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366455 2023-10-03T20:54:15.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366455 -le 1696367755 ]] 2023-10-03T20:54:15.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:54:15.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:54:15.402 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:54:17.041 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:17.018+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:17.056 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:17.054+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:54:18.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:18.786+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:54:20.036 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:20.022+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:54:20.036 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:20.026+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:54:20.036 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:20.026+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:54:20.036 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:20.030+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:23.663 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:23.662+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:54:23.675 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:23.670+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:54:23.675 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:23.670+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:54:23.682 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:23.674+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:54:23.687 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:23.682+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:54:23.688 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:23.682+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:23.901 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:23.898+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:54:23.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:23.898+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:54:23.909 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:23.906+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:23.927 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:23.922+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:54:23.927 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:23.922+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:54:23.928 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:23.922+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:54:24.186 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:24.182+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:25.413 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:54:25.413 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:54:25.418 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:54:25.423 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:54:26.094 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:26.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:54:26.095 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:26.093+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:54:26.106 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:26.101+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:26.755 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:54:26.762 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:54:26.763 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366466 2023-10-03T20:54:26.764 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366466 -le 1696367755 ]] 2023-10-03T20:54:26.764 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:54:26.764 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:54:26.764 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:54:36.775 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:54:36.782 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:54:36.794 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:54:36.796 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:54:38.197 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:54:38.211 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:54:38.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366478 2023-10-03T20:54:38.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366478 -le 1696367755 ]] 2023-10-03T20:54:38.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:54:38.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:54:38.212 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:54:38.920 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:38.917+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:40.366 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:40.357+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:54:40.366 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:40.357+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:54:40.372 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:40.365+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:40.374 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:40.369+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:54:40.374 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:40.369+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:54:43.887 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.881+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:54:43.904 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.901+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:54:43.904 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.901+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:54:43.904 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.901+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:54:43.904 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.901+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:54:43.912 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.905+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:54:43.913 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:43.905+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:47.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:47.897+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:54:47.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:47.897+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:54:47.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:47.909+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:47.923 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:47.921+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:54:47.924 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:47.921+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:54:47.941 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:47.937+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:54:48.216 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:54:48.216 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:54:48.218 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:54:48.221 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:54:49.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:54:49.503 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:54:49.509 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366489 2023-10-03T20:54:49.509 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366489 -le 1696367755 ]] 2023-10-03T20:54:49.509 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:54:49.509 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:54:49.509 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:54:50.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:50.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:54:50.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:54:50.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:54:51.559 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:54:51.553+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:54:59.516 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:54:59.516 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:54:59.516 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:54:59.519 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:55:00.911 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:55:00.911 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:55:00.911 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366500 2023-10-03T20:55:00.912 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366500 -le 1696367755 ]] 2023-10-03T20:55:00.912 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:55:00.912 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:55:00.912 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:55:01.903 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.897+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:55:01.903 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.897+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:55:01.909 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.905+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:55:01.918 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.913+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:55:01.923 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.921+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:55:01.923 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.921+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:55:01.932 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:01.925+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:55:02.751 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:02.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:02.751 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:02.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:55:02.751 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:02.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:55:02.762 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:02.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:02.778 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:02.761+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:03.252 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:03.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:55:03.254 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:03.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:04.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:55:04.345 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:04.337+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:04.345 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:04.337+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:55:04.345 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:04.341+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:04.357 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:04.353+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:10.918 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:55:10.918 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:55:10.918 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:55:10.929 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:55:12.188 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:55:12.192 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:55:12.193 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366512 2023-10-03T20:55:12.193 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366512 -le 1696367755 ]] 2023-10-03T20:55:12.193 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:55:12.194 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:55:12.194 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:55:16.059 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:16.057+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:16.109 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:16.101+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:19.307 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:19.301+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:19.307 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:19.301+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:19.307 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:19.301+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:22.198 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:55:22.212 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:55:22.217 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:55:22.223 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:55:23.242 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:55:23.246 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:55:23.247 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366523 2023-10-03T20:55:23.247 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366523 -le 1696367755 ]] 2023-10-03T20:55:23.247 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:55:23.247 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:55:23.247 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:55:28.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:28.880+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:55:28.892 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:55:28.884+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:55:33.260 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:55:33.262 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:55:33.264 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:55:33.281 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:55:34.195 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:55:34.202 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:55:34.211 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366534 2023-10-03T20:55:34.211 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366534 -le 1696367755 ]] 2023-10-03T20:55:34.211 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:55:34.211 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:55:34.211 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:55:39.268 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.252+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:39.282 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.272+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:39.404 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.400+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:39.426 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.416+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:55:39.426 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.416+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:55:39.427 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.416+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:39.427 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.416+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:39.427 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.416+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:39.711 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:39.704+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:44.234 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:55:44.240 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:55:44.241 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:55:44.279 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:55:45.602 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:55:45.602 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:55:45.603 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366545 2023-10-03T20:55:45.603 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366545 -le 1696367755 ]] 2023-10-03T20:55:45.603 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:55:45.603 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:55:45.603 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:55:45.798 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:45.792+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:45.799 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:45.792+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:55:45.799 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:45.792+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:55:45.812 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:45.808+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:45.812 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:45.808+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:45.812 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:45.808+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:46.538 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:46.532+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:46.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:46.560+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:46.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:46.560+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:46.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:46.560+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:47.287 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:47.280+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:47.291 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:47.284+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:47.291 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:47.288+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:49.601 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.596+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:55:49.606 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.600+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:49.606 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.600+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:55:49.606 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.600+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:55:49.611 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.604+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:49.623 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:49.623 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:49.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:54.464 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:54.456+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:55:54.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:54.472+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:55:54.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:54.472+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:55:54.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:54.492+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:55:54.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:54.492+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:55:54.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:55:54.492+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:55:55.605 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:55:55.613 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:55:55.615 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:55:55.627 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:55:56.868 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:55:56.896 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:55:56.901 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366556 2023-10-03T20:55:56.901 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366556 -le 1696367755 ]] 2023-10-03T20:55:56.902 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:55:56.902 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:55:56.902 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:56:06.907 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:56:06.907 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:56:06.908 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:56:06.919 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:56:08.419 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:56:08.419 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:56:08.420 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366568 2023-10-03T20:56:08.420 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366568 -le 1696367755 ]] 2023-10-03T20:56:08.420 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:56:08.420 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:56:08.420 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:56:11.451 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:11.436+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:56:11.462 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:11.456+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:11.462 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:11.456+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:11.463 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:11.456+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:11.463 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:11.456+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:11.580 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:11.576+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:56:11.987 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:11.980+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:12.021 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:12.016+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:12.022 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:12.016+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:14.349 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:14.344+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:56:14.361 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:14.344+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:14.387 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:14.380+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:56:15.423 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.416+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:56:15.441 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.432+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:15.441 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.432+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:15.441 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.432+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:56:15.441 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.436+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:15.452 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.448+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:15.452 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:15.448+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:18.423 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:56:18.424 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:56:18.430 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:56:18.458 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:56:19.783 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:56:19.783 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:56:19.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366579 2023-10-03T20:56:19.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366579 -le 1696367755 ]] 2023-10-03T20:56:19.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:56:19.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:56:19.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:56:23.833 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:23.828+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:23.849 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:23.844+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:23.859 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:23.856+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:26.151 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.144+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:56:26.159 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.152+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:26.159 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.152+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:26.159 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.152+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:56:26.159 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.152+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:26.159 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.152+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:26.163 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:26.156+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:26.268 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:26.264+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:29.797 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:56:29.797 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:56:29.798 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:56:29.798 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:56:30.827 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:30.819+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:30.836 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:30.827+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:56:31.089 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:56:31.091 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:56:31.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366591 2023-10-03T20:56:31.109 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366591 -le 1696367755 ]] 2023-10-03T20:56:31.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:56:31.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:56:31.110 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:56:32.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:32.375+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:56:32.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:32.375+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:56:32.386 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:32.379+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:32.396 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:32.391+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:56:34.016 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:34.011+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:34.016 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:34.011+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:34.028 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:34.023+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:34.055 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:34.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:34.055 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:34.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:34.922 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.915+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:56:34.934 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.927+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T20:56:34.935 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.927+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T20:56:34.935 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.927+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:56:34.935 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.927+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:56:34.939 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.935+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:34.939 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.935+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:56:34.939 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.935+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:56:34.939 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.935+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:56:34.939 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:34.935+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T20:56:41.121 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:56:41.121 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:56:41.122 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:56:41.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:56:42.294 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.287+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:56:42.304 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:42.304 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:42.304 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:56:42.304 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:42.304 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:42.304 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:42.299+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:42.528 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:56:42.528 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:56:42.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366602 2023-10-03T20:56:42.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366602 -le 1696367755 ]] 2023-10-03T20:56:42.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:56:42.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:56:42.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:56:51.104 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:51.099+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:56:51.109 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:51.103+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:51.138 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:51.131+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:56:52.532 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:56:52.533 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:56:52.533 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:56:52.535 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:56:53.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:56:53.780 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:56:53.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366613 2023-10-03T20:56:53.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366613 -le 1696367755 ]] 2023-10-03T20:56:53.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:56:53.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:56:53.780 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:56:54.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.831+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:56:54.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:54.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:54.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:56:54.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.843+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:54.856 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.851+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:54.857 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:54.851+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:54.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:56:54.967+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:56:55.387 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:55.379+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:55.393 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:55.387+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:55.393 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:55.387+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:56:59.564 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:59.559+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:56:59.569 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:59.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:56:59.579 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:59.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:56:59.579 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:59.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:56:59.581 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:59.575+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:56:59.591 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:56:59.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:57:03.797 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:57:03.799 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:57:03.803 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:57:03.807 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:57:04.480 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:04.475+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:57:04.882 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:57:04.882 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:57:04.885 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366624 2023-10-03T20:57:04.885 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366624 -le 1696367755 ]] 2023-10-03T20:57:04.886 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:57:04.886 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:57:04.886 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:57:12.231 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:12.219+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:57:12.232 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:12.219+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:57:12.232 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:12.219+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:57:12.232 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:12.219+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:57:14.896 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:57:14.900 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:57:14.918 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:57:14.937 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:57:16.175 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:57:16.182 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:57:16.183 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366636 2023-10-03T20:57:16.183 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366636 -le 1696367755 ]] 2023-10-03T20:57:16.183 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:57:16.183 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:57:16.183 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:57:26.185 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:57:26.185 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:57:26.198 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:57:26.200 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:57:27.398 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:57:27.398 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:57:27.399 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366647 2023-10-03T20:57:27.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366647 -le 1696367755 ]] 2023-10-03T20:57:27.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:57:27.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:57:27.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:57:37.423 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:57:37.423 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:57:37.435 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:57:37.437 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:57:38.437 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:38.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:57:38.437 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:38.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:57:38.438 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:38.430+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:57:38.443 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:38.438+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:57:38.444 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:38.438+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:57:38.444 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:38.438+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:57:38.747 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:57:38.748 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:57:38.750 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366658 2023-10-03T20:57:38.750 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366658 -le 1696367755 ]] 2023-10-03T20:57:38.750 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:57:38.750 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:57:38.750 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:57:41.983 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:41.978+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:57:41.983 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:41.978+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:57:41.989 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:41.982+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:57:41.993 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:41.986+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:57:42.001 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:41.994+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:57:42.034 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:42.026+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:57:44.451 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.442+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:57:44.466 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.458+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:57:44.887 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.878+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:57:44.908 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:57:44.909 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:57:44.909 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:57:44.934 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.926+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:57:44.935 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:44.926+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:57:48.767 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:57:48.768 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:57:48.768 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:57:48.774 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:57:50.056 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:57:50.057 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:57:50.057 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366670 2023-10-03T20:57:50.058 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366670 -le 1696367755 ]] 2023-10-03T20:57:50.058 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:57:50.058 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:57:50.058 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:57:52.596 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:52.590+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:57:52.596 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:52.590+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:57:52.596 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:52.590+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:57:52.596 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:57:52.590+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:57:55.376 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:55.362+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:57:55.377 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:55.362+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:57:55.377 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:55.362+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:57:56.781 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:56.762+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:57:56.782 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:56.770+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:57:56.788 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:57:56.778+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:58:00.065 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:58:00.067 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:58:00.067 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:58:00.077 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:58:01.559 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:58:01.560 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:58:01.560 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366681 2023-10-03T20:58:01.560 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366681 -le 1696367755 ]] 2023-10-03T20:58:01.560 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:58:01.560 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:58:01.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:58:01.988 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:01.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:01.988 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:01.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:01.988 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:01.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:58:01.988 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:01.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:58:02.200 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:02.194+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:02.213 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:02.202+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:02.245 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:02.238+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:06.574 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:06.566+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:58:11.562 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:58:11.562 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:58:11.573 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:58:11.574 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:58:12.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:12.318+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:58:12.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:12.326+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:12.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:12.326+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:12.627 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:12.622+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:12.664 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:12.638+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:13.042 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:58:13.043 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:58:13.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366693 2023-10-03T20:58:13.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366693 -le 1696367755 ]] 2023-10-03T20:58:13.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:58:13.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:58:13.044 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:58:23.049 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:58:23.049 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:58:23.050 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:58:23.061 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:58:23.405 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:23.398+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:58:24.123 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:58:24.124 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:58:24.124 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366704 2023-10-03T20:58:24.125 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366704 -le 1696367755 ]] 2023-10-03T20:58:24.125 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:58:24.125 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:58:24.125 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:58:33.709 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:33.706+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:58:34.138 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:58:34.138 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:58:34.148 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:58:34.157 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:58:35.536 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:58:35.537 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:58:35.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366715 2023-10-03T20:58:35.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366715 -le 1696367755 ]] 2023-10-03T20:58:35.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:58:35.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:58:35.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:58:37.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:37.254+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:37.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:37.254+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:37.259 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:37.254+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:37.267 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:37.266+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:58:37.267 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:37.266+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:38.205 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.202+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:58:38.224 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.218+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:38.225 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.218+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:38.225 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.218+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:58:38.225 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.222+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:38.233 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.230+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:58:38.233 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:38.230+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:43.886 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:43.881+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:58:43.887 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:43.881+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:58:43.898 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:43.893+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:58:43.916 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:43.913+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:58:43.917 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:43.913+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:58:43.931 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:58:43.929+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:58:44.469 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.461+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:58:44.472 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.469+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:44.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:44.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:58:44.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:44.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:58:44.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:44.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:45.542 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:58:45.542 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:58:45.543 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:58:45.553 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:58:46.731 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:58:46.743 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:58:46.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366726 2023-10-03T20:58:46.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366726 -le 1696367755 ]] 2023-10-03T20:58:46.746 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:58:46.747 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:58:46.747 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:58:53.115 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:53.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:58:53.130 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:53.125+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:53.130 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:53.125+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:53.140 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:53.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:53.142 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:53.141+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:58:53.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:53.141+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:56.759 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:58:56.759 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:58:56.769 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:58:56.772 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:58:57.883 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:57.873+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:57.904 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:57.901+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:58.122 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:58:58.122 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:58:58.123 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366738 2023-10-03T20:58:58.124 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366738 -le 1696367755 ]] 2023-10-03T20:58:58.124 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:58:58.124 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:58:58.124 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:58:58.346 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.341+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:58:58.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.349+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:58.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.349+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:58.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.349+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:58:58.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.349+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:58:58.358 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.349+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:58:58.359 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:58.349+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:58:59.208 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:59.205+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:58:59.208 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:59.205+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:58:59.208 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:59.205+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:58:59.251 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:58:59.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:59:00.279 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:00.277+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:59:00.279 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:00.277+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:59:04.141 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:04.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:59:04.141 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:04.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:59:04.141 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:04.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:59:04.141 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:04.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:59:05.855 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.853+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:59:05.856 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.853+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:59:05.865 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.861+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:59:05.885 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.881+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:59:05.885 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.881+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:59:05.885 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.881+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T20:59:05.885 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:05.881+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:59:08.134 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:59:08.140 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:59:08.141 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:59:08.141 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:59:09.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:59:09.495 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:59:09.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366749 2023-10-03T20:59:09.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366749 -le 1696367755 ]] 2023-10-03T20:59:09.495 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:59:09.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:59:09.496 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:59:10.360 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:10.357+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:59:10.375 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:10.373+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:59:12.315 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.309+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T20:59:12.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:59:12.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:59:12.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.325+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:59:12.334 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:59:12.334 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:59:12.334 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:12.329+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:59:17.928 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:17.925+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T20:59:17.928 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:17.925+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:59:19.503 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:59:19.505 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:59:19.521 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:59:19.530 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:59:20.719 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:59:20.719 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:59:20.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366760 2023-10-03T20:59:20.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366760 -le 1696367755 ]] 2023-10-03T20:59:20.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:59:20.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:59:20.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:59:28.937 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:28.933+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:59:28.937 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:28.933+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:59:28.940 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:28.933+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:59:28.948 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:28.945+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:59:28.948 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:28.945+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:59:30.723 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:59:30.723 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:59:30.730 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:59:30.735 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:59:32.058 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:59:32.061 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:59:32.062 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366772 2023-10-03T20:59:32.063 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366772 -le 1696367755 ]] 2023-10-03T20:59:32.063 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:59:32.063 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:59:32.063 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:59:35.436 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.433+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:59:35.436 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.433+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:59:35.906 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.901+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T20:59:35.906 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.901+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T20:59:35.919 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.909+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:59:35.919 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.913+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T20:59:35.919 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.913+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T20:59:35.933 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:35.929+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T20:59:38.387 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:38.385+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T20:59:38.388 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:38.385+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T20:59:38.388 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:38.385+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T20:59:38.389 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:38.385+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T20:59:38.390 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:38.385+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T20:59:38.390 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T20:59:38.385+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T20:59:42.083 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:59:42.089 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:59:42.090 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:59:42.113 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:59:43.036 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:59:43.037 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:59:43.037 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366783 2023-10-03T20:59:43.037 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366783 -le 1696367755 ]] 2023-10-03T20:59:43.037 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:59:43.038 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:59:43.038 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T20:59:47.742 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T20:59:47.736+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T20:59:53.043 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T20:59:53.048 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T20:59:53.058 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T20:59:53.059 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T20:59:54.159 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T20:59:54.159 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T20:59:54.160 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366794 2023-10-03T20:59:54.160 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366794 -le 1696367755 ]] 2023-10-03T20:59:54.160 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T20:59:54.160 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T20:59:54.160 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:00:02.772 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:00:02.772 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.768+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:02.843 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.840+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:02.843 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.840+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:00:02.843 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.840+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:00:02.843 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.840+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:00:02.843 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.840+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:00:02.849 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:02.844+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:00:04.163 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:00:04.163 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:00:04.165 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:00:04.175 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:00:05.110 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:05.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:00:05.110 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:05.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:00:05.110 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:05.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:00:05.110 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:05.104+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:05.110 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:05.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:00:05.110 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:05.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:00:05.331 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:00:05.332 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:00:05.332 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366805 2023-10-03T21:00:05.332 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366805 -le 1696367755 ]] 2023-10-03T21:00:05.332 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:00:05.333 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:00:05.333 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:00:07.245 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.240+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:07.245 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.240+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:00:07.245 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.240+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:00:07.245 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.240+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:07.245 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.240+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:07.537 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.532+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:07.549 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.544+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:00:07.549 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:07.544+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:15.363 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:00:15.368 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:00:15.371 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:00:15.375 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:00:16.753 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:00:16.754 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:00:16.755 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366816 2023-10-03T21:00:16.756 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366816 -le 1696367755 ]] 2023-10-03T21:00:16.756 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:00:16.756 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:00:16.756 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:00:17.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:17.648+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:17.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:17.648+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:00:17.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:17.648+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:00:17.654 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:17.648+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:00:18.665 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:18.652+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:18.670 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:18.664+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:25.521 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:25.516+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:26.764 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:00:26.764 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:00:26.780 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:00:26.799 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:00:27.866 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:27.852+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:27.867 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:27.852+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:00:27.867 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:27.852+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:00:27.867 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:27.852+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:27.867 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:27.852+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:28.039 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:00:28.039 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:00:28.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366828 2023-10-03T21:00:28.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366828 -le 1696367755 ]] 2023-10-03T21:00:28.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:00:28.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:00:28.040 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:00:37.511 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:37.508+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:38.063 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:00:38.063 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:00:38.063 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:00:38.075 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:00:39.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:00:39.367 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:00:39.368 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366839 2023-10-03T21:00:39.368 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366839 -le 1696367755 ]] 2023-10-03T21:00:39.368 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:00:39.368 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:00:39.368 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:00:41.107 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:41.103+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:42.564 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:42.559+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:42.580 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:42.575+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:42.625 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:42.619+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:00:42.625 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:42.619+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:00:43.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:43.635+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:45.326 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:45.319+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:45.355 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:45.351+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:00:45.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:45.351+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:47.285 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:47.279+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:00:47.286 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:47.279+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:47.292 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:47.279+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:49.378 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:00:49.378 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:00:49.381 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:00:49.382 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:00:50.493 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:00:50.493 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:00:50.493 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366850 2023-10-03T21:00:50.493 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366850 -le 1696367755 ]] 2023-10-03T21:00:50.493 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:00:50.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:00:50.494 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:00:50.594 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.587+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:50.594 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.587+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:00:50.614 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.607+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:50.622 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.615+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:00:50.634 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.627+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:50.768 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.755+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:00:50.777 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:50.777 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:00:50.777 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:00:50.777 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:50.777 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:50.771+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:54.216 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:54.203+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:54.217 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:54.203+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:00:54.235 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:54.231+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:00:54.236 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:54.231+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:56.602 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:56.595+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:00:56.628 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:00:56.623+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:00:57.664 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:57.659+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:00:57.664 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:57.659+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:00:57.664 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:57.659+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:57.664 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:57.659+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:00:57.664 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:57.659+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:00:58.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.879+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:00:58.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.879+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:00:58.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.879+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:00:58.893 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.887+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:00:58.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.895+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:00:58.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.895+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:00:58.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.895+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:00:58.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:00:58.895+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:01:00.498 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:01:00.498 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:01:00.506 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:01:00.512 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:01:01.258 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:01.251+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:01:01.260 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:01.251+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:01:01.276 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:01.267+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:01:01.675 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:01:01.676 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:01:01.678 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366861 2023-10-03T21:01:01.678 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366861 -le 1696367755 ]] 2023-10-03T21:01:01.678 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:01:01.678 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:01:01.678 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:01:04.281 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:04.271+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:01:06.212 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:06.207+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:01:06.213 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:06.207+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:01:06.217 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:06.211+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:01:06.224 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:06.219+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:01:06.224 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:06.219+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:01:08.536 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:08.511+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:01:08.555 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:08.547+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:01:11.699 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:01:11.700 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:01:11.700 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:01:11.706 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:01:13.021 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:01:13.022 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:01:13.023 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366873 2023-10-03T21:01:13.023 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366873 -le 1696367755 ]] 2023-10-03T21:01:13.023 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:01:13.024 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:01:13.024 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:01:13.562 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:13.547+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:01:13.563 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:13.547+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:01:13.564 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:13.559+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:01:13.569 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:13.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:01:13.569 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:13.563+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:01:15.030 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:15.023+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:01:15.030 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:15.023+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:01:15.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:15.039+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:01:15.046 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:15.039+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:01:20.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:20.411+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:01:23.028 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:01:23.028 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:01:23.029 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:01:23.039 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:01:23.042 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.035+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:01:23.042 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.035+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:01:23.043 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.035+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:01:23.044 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.039+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:01:23.065 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.055+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:01:23.065 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.055+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:01:23.065 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.055+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:01:23.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.055+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:01:23.066 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:23.055+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:01:24.417 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:01:24.419 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:01:24.421 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366884 2023-10-03T21:01:24.422 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366884 -le 1696367755 ]] 2023-10-03T21:01:24.422 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:01:24.422 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:01:24.422 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:01:26.370 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:26.362+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:01:26.387 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:26.378+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:01:26.388 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:01:26.382+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:01:27.930 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:27.918+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:01:27.930 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:27.922+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:01:29.084 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:29.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:01:29.111 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:29.102+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:01:34.443 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:01:34.450 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:01:34.459 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:01:34.459 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:01:35.686 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:01:35.693 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:01:35.695 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366895 2023-10-03T21:01:35.695 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366895 -le 1696367755 ]] 2023-10-03T21:01:35.695 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:01:35.695 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:01:35.695 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:01:45.705 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:01:45.710 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:01:45.727 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:01:45.727 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:01:47.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:01:47.280 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:01:47.295 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366907 2023-10-03T21:01:47.295 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366907 -le 1696367755 ]] 2023-10-03T21:01:47.295 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:01:47.296 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:01:47.296 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:01:47.373 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:47.366+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:01:52.158 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:01:52.150+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:01:57.312 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:01:57.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:01:57.319 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:01:57.331 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:01:58.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:01:58.852 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:01:58.853 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366918 2023-10-03T21:01:58.853 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366918 -le 1696367755 ]] 2023-10-03T21:01:58.853 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:01:58.853 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:01:58.853 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:02:08.766 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.762+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:02:08.767 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.762+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:02:08.788 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.786+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:02:08.805 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.802+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:02:08.805 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.802+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:02:08.805 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.802+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:02:08.806 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.802+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:02:08.823 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:08.822+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:02:08.855 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:02:08.863 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:02:08.866 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:02:08.867 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:02:10.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:02:10.169 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:02:10.170 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366930 2023-10-03T21:02:10.170 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366930 -le 1696367755 ]] 2023-10-03T21:02:10.170 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:02:10.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:02:10.171 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:02:16.961 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:16.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:02:20.189 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:02:20.191 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:02:20.192 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:02:20.196 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:02:21.531 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:02:21.532 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:02:21.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366941 2023-10-03T21:02:21.538 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366941 -le 1696367755 ]] 2023-10-03T21:02:21.539 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:02:21.539 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:02:21.539 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:02:24.907 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:24.905+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:02:24.907 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:24.905+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:02:24.909 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:24.905+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:02:24.920 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:24.917+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:02:24.920 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:24.917+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:02:24.920 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:24.917+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:02:31.449 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:31.445+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:02:31.554 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:02:31.554 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:02:31.555 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:02:31.563 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:02:32.782 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:02:32.782 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:02:32.783 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366952 2023-10-03T21:02:32.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366952 -le 1696367755 ]] 2023-10-03T21:02:32.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:02:32.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:02:32.784 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:02:37.691 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:37.685+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:02:37.692 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:37.685+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:02:41.684 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:41.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:02:41.684 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:41.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:02:41.684 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:41.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:02:41.684 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:41.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:02:41.684 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:41.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:02:41.684 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:02:41.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:02:42.797 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:02:42.798 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:02:42.811 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:02:42.813 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:02:43.990 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:02:43.990 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:02:43.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366963 2023-10-03T21:02:43.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366963 -le 1696367755 ]] 2023-10-03T21:02:43.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:02:43.992 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:02:43.992 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:02:44.530 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.525+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:02:44.545 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.541+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:02:44.545 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.541+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:02:44.547 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.545+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:02:44.554 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.549+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:02:44.554 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.549+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:02:44.562 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:02:44.557+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:02:54.003 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:02:54.003 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:02:54.003 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:02:54.008 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:02:55.435 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:02:55.435 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:02:55.451 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366975 2023-10-03T21:02:55.452 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366975 -le 1696367755 ]] 2023-10-03T21:02:55.452 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:02:55.452 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:02:55.452 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:03:05.453 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:03:05.453 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:03:05.467 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:03:05.475 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:03:06.827 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:03:06.827 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:03:06.829 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366986 2023-10-03T21:03:06.829 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366986 -le 1696367755 ]] 2023-10-03T21:03:06.829 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:03:06.829 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:03:06.829 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:03:09.901 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:09.896+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:03:10.506 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:10.496+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:03:10.507 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:10.500+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:03:10.507 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:10.500+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:03:10.513 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:10.508+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:03:10.523 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:10.516+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:03:10.523 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:10.516+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:03:15.628 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:15.624+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:03:15.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:15.644+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:03:16.851 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:03:16.862 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:03:16.867 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:03:16.879 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:03:18.103 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:03:18.106 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:03:18.107 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696366998 2023-10-03T21:03:18.107 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696366998 -le 1696367755 ]] 2023-10-03T21:03:18.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:03:18.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:03:18.108 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:03:21.114 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:03:21.114 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:03:21.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:03:21.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:03:21.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:03:21.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:03:21.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:03:21.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:21.104+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:03:23.148 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:23.144+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:03:23.161 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:23.156+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:03:27.489 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.480+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:03:27.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.488+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:03:27.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.488+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:03:27.496 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.488+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:03:27.501 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.496+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:03:27.507 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.500+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:03:27.507 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:27.500+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:03:28.112 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:03:28.113 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:03:28.114 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:03:28.125 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:03:29.393 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:03:29.398 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:03:29.399 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367009 2023-10-03T21:03:29.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367009 -le 1696367755 ]] 2023-10-03T21:03:29.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:03:29.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:03:29.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:03:31.154 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:31.148+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:03:32.131 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:32.124+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:03:32.142 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:32.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:03:32.142 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:32.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:03:32.142 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:32.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:03:32.142 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:32.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:03:32.142 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:32.136+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:03:35.295 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:35.292+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:03:35.306 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:35.300+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:03:35.306 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:35.300+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:03:39.412 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:03:39.415 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:03:39.428 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:03:39.437 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:03:40.715 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:03:40.715 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:03:40.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367020 2023-10-03T21:03:40.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367020 -le 1696367755 ]] 2023-10-03T21:03:40.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:03:40.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:03:40.720 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:03:48.074 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:48.068+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:03:50.733 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:03:50.733 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:03:50.750 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:03:50.765 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:03:50.916 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:50.912+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:03:50.916 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:50.912+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:03:50.947 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:50.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:03:50.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:50.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:03:50.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:50.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:03:50.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:03:50.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:03:51.946 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:03:51.946 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:03:51.948 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367031 2023-10-03T21:03:51.948 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367031 -le 1696367755 ]] 2023-10-03T21:03:51.948 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:03:51.948 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:03:51.948 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:03:54.703 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:54.700+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:03:54.714 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:54.708+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:03:58.305 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:03:58.300+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:04:01.951 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:04:01.959 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:04:01.970 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:04:01.970 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:04:03.409 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:04:03.409 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:04:03.410 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367043 2023-10-03T21:04:03.410 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367043 -le 1696367755 ]] 2023-10-03T21:04:03.410 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:04:03.410 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:04:03.410 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:04:05.772 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:05.767+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:04:05.779 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:05.775+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:04:11.356 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:11.343+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:04:11.357 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:11.351+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:04:11.357 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:11.351+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:04:11.375 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:11.367+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:04:11.377 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:11.367+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:04:11.377 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:11.367+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:04:13.412 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:04:13.412 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:04:13.414 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:04:13.420 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:04:14.883 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:04:14.884 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:04:14.884 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367054 2023-10-03T21:04:14.885 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367054 -le 1696367755 ]] 2023-10-03T21:04:14.885 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:04:14.885 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:04:14.885 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:04:16.408 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.399+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:04:16.408 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.399+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:04:16.408 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.399+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:04:16.418 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.411+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:04:16.423 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.419+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:04:16.423 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.419+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:04:16.424 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.419+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:04:16.424 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:04:16.419+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:04:24.891 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:04:24.891 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:04:24.891 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:04:24.925 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:04:26.099 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:04:26.099 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:04:26.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367066 2023-10-03T21:04:26.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367066 -le 1696367755 ]] 2023-10-03T21:04:26.101 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:04:26.101 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:04:26.101 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:04:36.111 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:04:36.111 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:04:36.111 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:04:36.113 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:04:37.223 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:04:37.224 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:04:37.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367077 2023-10-03T21:04:37.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367077 -le 1696367755 ]] 2023-10-03T21:04:37.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:04:37.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:04:37.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:04:38.046 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:38.039+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:04:38.075 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:38.067+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:04:38.075 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:38.067+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:04:38.075 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:38.067+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:04:40.786 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:40.779+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:04:40.786 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:40.779+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:04:40.793 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:40.787+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:04:40.802 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:40.791+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:04:40.802 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:40.795+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:04:42.539 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:42.535+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:04:42.554 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:42.547+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:04:42.554 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:42.547+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:04:47.226 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:04:47.231 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:04:47.247 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:04:47.263 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:04:48.631 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:04:48.639 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:04:48.647 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367088 2023-10-03T21:04:48.647 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367088 -le 1696367755 ]] 2023-10-03T21:04:48.647 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:04:48.647 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:04:48.647 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:04:50.869 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:50.863+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:04:50.869 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:50.863+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:04:50.881 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:50.875+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:04:50.907 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:50.903+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:04:55.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:55.827+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:04:55.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:55.827+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:04:55.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:55.827+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:04:55.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:04:55.827+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:04:58.689 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:04:58.696 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:04:58.696 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:04:58.708 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:05:00.358 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:05:00.358 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:05:00.359 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367100 2023-10-03T21:05:00.359 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367100 -le 1696367755 ]] 2023-10-03T21:05:00.359 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:05:00.359 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:05:00.359 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:05:01.316 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:05:01.310+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:05:01.328 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:05:01.322+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:05:02.887 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:02.874+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:02.891 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:02.882+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:02.904 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:02.898+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:05.594 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:05:05.586+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:05:07.863 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:07.858+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:10.366 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:05:10.371 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:05:10.371 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:05:10.372 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:05:11.375 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:05:11.375 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:05:11.375 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367111 2023-10-03T21:05:11.376 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367111 -le 1696367755 ]] 2023-10-03T21:05:11.376 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:05:11.376 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:05:11.376 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:05:20.350 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:20.342+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:05:20.357 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:20.350+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:20.357 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:20.350+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:20.362 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:20.354+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:20.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:20.362+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:20.369 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:20.362+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:21.380 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:05:21.380 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:05:21.386 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:05:21.387 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:05:22.993 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:05:22.993 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:05:22.994 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367122 2023-10-03T21:05:22.994 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367122 -le 1696367755 ]] 2023-10-03T21:05:22.994 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:05:22.994 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:05:22.994 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:05:25.535 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:25.526+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:25.535 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:25.526+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:25.537 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:25.530+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:25.557 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:25.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:25.557 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:25.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:27.527 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:27.510+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:05:27.528 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:27.514+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:27.551 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:27.530+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:27.561 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:27.554+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:27.561 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:27.554+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:32.831 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:32.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:05:32.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:32.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:32.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:32.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:32.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:32.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:05:32.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:32.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:32.850 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:32.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:33.024 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:05:33.024 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:05:33.025 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:05:33.029 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:05:34.095 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:05:34.086+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:05:34.142 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:05:34.142 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:05:34.142 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367134 2023-10-03T21:05:34.142 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367134 -le 1696367755 ]] 2023-10-03T21:05:34.143 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:05:34.143 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:05:34.143 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:05:35.059 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.050+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:05:35.065 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:35.077 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:35.077 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:05:35.078 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:35.084 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:35.085 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:35.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:39.857 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:39.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:39.858 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:39.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:39.858 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:39.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:39.865 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:39.858+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:39.865 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:39.858+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:41.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:41.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:05:41.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:41.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:41.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:41.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:41.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:41.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:05:41.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:41.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:41.832 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:41.826+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:44.152 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:05:44.152 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:05:44.161 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:05:44.184 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:05:44.257 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:44.254+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:44.257 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:44.254+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:44.262 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:44.258+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:44.269 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:44.266+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:44.269 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:44.266+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:45.338 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:05:45.338 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:05:45.339 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367145 2023-10-03T21:05:45.339 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367145 -le 1696367755 ]] 2023-10-03T21:05:45.339 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:05:45.339 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:05:45.339 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:05:49.945 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:49.942+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:05:49.953 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:49.950+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:49.953 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:49.950+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:05:49.956 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:49.954+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:49.979 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:49.978+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:49.984 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:49.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:50.474 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:50.470+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:05:50.491 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:50.486+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:05:50.495 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:50.490+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:05:50.495 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:05:50.490+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:05:55.349 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:05:55.350 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:05:55.350 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:05:55.366 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:05:56.809 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:05:56.809 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:05:56.810 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367156 2023-10-03T21:05:56.810 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367156 -le 1696367755 ]] 2023-10-03T21:05:56.810 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:05:56.810 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:05:56.810 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:06:01.778 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:01.773+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:06:01.779 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:01.773+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:06:01.786 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:01.781+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:02.120 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:02.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:02.131 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:02.129+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:02.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:02.141+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:06:02.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:02.141+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:04.104 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:04.101+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:06:04.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:04.101+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:06:04.112 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:04.109+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:04.123 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:04.117+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:06:04.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:04.121+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:06:04.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:04.121+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:06:06.812 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:06:06.812 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:06:06.816 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:06:06.819 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:06:07.969 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:06:07.969 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:06:07.974 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367167 2023-10-03T21:06:07.974 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367167 -le 1696367755 ]] 2023-10-03T21:06:07.974 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:06:07.975 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:06:07.975 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:06:10.625 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:10.621+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:10.652 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:10.649+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:06:10.666 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:10.661+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:13.231 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:13.225+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:06:13.243 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:13.237+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:13.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:13.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:13.258 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:13.253+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:13.258 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:13.253+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:06:13.258 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:13.253+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:17.761 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:17.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:17.770 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:17.765+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:17.975 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:06:17.976 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:06:17.977 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:06:17.994 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:06:18.893 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:18.889+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:18.897 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:18.893+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:19.247 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:06:19.251 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:06:19.254 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367179 2023-10-03T21:06:19.254 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367179 -le 1696367755 ]] 2023-10-03T21:06:19.254 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:06:19.254 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:06:19.255 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:06:19.675 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:19.669+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:06:19.680 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:19.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:19.680 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:19.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:06:19.681 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:19.677+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:06:19.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:19.701+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:19.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:19.701+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:06:19.875 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:19.865+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:24.425 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:24.421+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:26.087 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:26.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:26.088 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:26.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:06:26.088 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:26.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:06:26.103 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:26.093+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:26.103 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:26.097+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:06:26.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:26.105+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:29.256 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:06:29.257 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:06:29.257 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:06:29.276 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:06:30.768 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:06:30.771 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:06:30.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367190 2023-10-03T21:06:30.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367190 -le 1696367755 ]] 2023-10-03T21:06:30.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:06:30.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:06:30.779 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:06:31.410 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:31.405+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:31.410 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:31.405+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:06:31.429 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:31.421+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:31.429 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:31.421+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:06:31.437 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:31.433+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:39.000 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:38.997+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:40.780 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:06:40.781 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:06:40.782 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:06:40.784 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:06:42.193 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:06:42.194 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:06:42.195 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367202 2023-10-03T21:06:42.195 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367202 -le 1696367755 ]] 2023-10-03T21:06:42.195 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:06:42.195 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:06:42.195 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:06:47.742 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.737+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:06:47.746 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.741+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:06:47.746 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.741+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:06:47.747 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.741+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:06:47.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:06:47.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.753+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:06:47.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:06:47.757+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:06:52.203 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:06:52.211 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:06:52.211 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:06:52.212 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:06:53.466 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:06:53.466 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:06:53.466 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367213 2023-10-03T21:06:53.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367213 -le 1696367755 ]] 2023-10-03T21:06:53.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:06:53.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:06:53.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:06:58.011 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:58.008+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:58.035 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:58.028+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:06:58.035 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:58.028+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:06:58.051 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:58.040+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:06:59.761 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:59.756+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:06:59.761 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:59.756+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:06:59.770 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:59.764+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:06:59.770 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:06:59.764+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:00.646 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:00.640+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:00.650 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:00.644+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:03.470 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:07:03.470 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:07:03.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:07:03.487 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:07:04.517 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:04.508+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:04.527 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:04.524+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:04.897 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:07:04.897 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:07:04.898 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367224 2023-10-03T21:07:04.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367224 -le 1696367755 ]] 2023-10-03T21:07:04.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:07:04.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:07:04.899 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:07:09.008 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:09.004+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:07:09.017 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:09.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:09.043 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:09.028+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:07:09.044 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:09.040+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:13.583 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:13.576+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:13.602 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:13.592+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:14.904 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:07:14.905 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:07:14.905 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:07:14.915 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:07:16.155 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:07:16.167 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:07:16.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367236 2023-10-03T21:07:16.168 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367236 -le 1696367755 ]] 2023-10-03T21:07:16.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:07:16.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:07:16.169 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:07:18.442 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:18.436+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:18.452 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:18.448+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:07:20.708 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:20.704+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:21.947 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.940+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.940+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.944+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:07:21.948 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:21.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:07:25.982 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:25.976+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:26.000 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:25.996+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:07:26.170 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:07:26.170 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:07:26.173 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:07:26.180 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:07:27.531 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:07:27.536 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:07:27.537 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367247 2023-10-03T21:07:27.537 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367247 -le 1696367755 ]] 2023-10-03T21:07:27.537 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:07:27.537 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:07:27.537 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:07:30.898 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:30.892+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:30.909 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:30.904+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:30.919 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:30.908+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:31.892 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:31.888+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:34.510 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:34.504+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:37.542 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:07:37.543 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:07:37.564 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:07:37.577 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:07:37.896 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:37.888+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:37.896 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:37.888+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:38.785 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:07:38.786 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:07:38.786 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367258 2023-10-03T21:07:38.786 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367258 -le 1696367755 ]] 2023-10-03T21:07:38.786 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:07:38.787 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:07:38.787 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:07:39.245 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:39.236+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:40.200 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:40.192+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:07:40.200 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:40.192+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:40.219 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:40.216+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:40.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:40.216+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:40.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:40.216+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:07:40.237 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:40.232+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:07:42.972 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:42.964+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:07:42.974 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:42.968+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:42.988 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:42.984+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:07:42.989 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:42.984+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:43.774 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:07:43.808 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.804+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:43.808 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.804+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:43.809 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.804+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:43.809 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.804+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:07:43.809 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.804+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:07:43.809 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:43.804+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:07:43.961 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:43.952+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:07:43.962 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:43.952+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:07:43.970 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:43.964+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:43.980 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:43.976+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:44.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:44.180+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:07:44.191 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:44.180+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:44.194 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:44.188+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:44.203 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:44.196+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:44.204 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:44.196+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:07:48.788 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:07:48.791 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:07:48.798 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:07:48.807 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:07:49.195 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:49.192+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:07:49.211 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:49.204+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:49.228 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:49.216+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:07:49.228 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:49.216+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:49.949 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:07:49.953 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:07:49.954 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367269 2023-10-03T21:07:49.954 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367269 -le 1696367755 ]] 2023-10-03T21:07:49.955 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:07:49.955 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:07:49.955 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:07:51.602 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:51.595+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:51.618 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:51.611+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:55.106 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.099+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:07:55.106 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.099+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:07:55.107 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.099+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:07:55.114 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:07:55.114 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:07:55.114 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:07:55.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:55.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:07:57.043 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.039+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:07:57.050 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.043+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:07:57.050 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.043+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:07:57.050 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.043+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:07:57.051 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.043+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:07:57.051 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.043+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:07:57.051 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:07:57.043+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:07:59.932 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:07:59.923+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:07:59.969 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:07:59.969 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:07:59.972 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:07:59.975 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:08:01.277 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:08:01.279 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:08:01.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367281 2023-10-03T21:08:01.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367281 -le 1696367755 ]] 2023-10-03T21:08:01.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:08:01.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:08:01.280 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:08:04.333 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:04.327+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:08:11.283 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:08:11.283 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:08:11.283 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:08:11.303 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:08:12.466 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:08:12.466 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:08:12.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367292 2023-10-03T21:08:12.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367292 -le 1696367755 ]] 2023-10-03T21:08:12.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:08:12.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:08:12.467 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:08:12.531 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:12.527+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:08:12.556 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:12.551+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:08:14.051 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:14.047+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:08:14.056 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:14.051+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:08:14.070 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:14.063+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:08:14.073 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:14.067+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:08:20.030 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:20.023+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:08:20.030 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:20.023+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:08:22.469 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:08:22.469 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:08:22.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:08:22.488 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:08:23.946 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:08:23.946 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:08:23.947 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367303 2023-10-03T21:08:23.947 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367303 -le 1696367755 ]] 2023-10-03T21:08:23.947 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:08:23.947 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:08:23.947 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:08:33.949 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:08:33.949 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:08:33.949 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:08:33.951 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:08:35.377 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:08:35.380 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:08:35.381 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367315 2023-10-03T21:08:35.381 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367315 -le 1696367755 ]] 2023-10-03T21:08:35.381 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:08:35.381 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:08:35.381 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:08:35.776 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.771+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:08:35.777 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.771+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:08:35.777 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.771+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:08:35.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:08:35.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:08:35.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:08:35.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:08:35.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:08:35.789 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:08:35.790 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:08:35.790 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:35.783+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:08:38.568 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:38.555+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:08:42.860 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:42.855+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:08:42.860 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:42.855+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:08:42.861 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:42.855+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:08:42.861 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:42.855+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:08:42.861 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:42.855+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:08:42.861 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:42.855+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:08:44.446 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:44.439+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:08:44.446 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:44.439+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:08:44.449 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:44.443+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:08:44.449 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:44.443+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:08:45.383 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:08:45.383 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:08:45.395 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:08:45.396 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:08:46.850 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:08:46.850 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:08:46.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367326 2023-10-03T21:08:46.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367326 -le 1696367755 ]] 2023-10-03T21:08:46.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:08:46.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:08:46.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:08:49.571 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:49.562+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:08:49.571 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:49.562+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:08:49.571 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:49.562+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:08:52.918 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:52.910+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:08:52.921 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:52.914+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:08:54.987 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:54.978+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:08:54.987 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:54.978+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:08:55.007 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:54.998+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:08:55.015 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:55.010+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:08:55.015 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:08:55.010+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:08:56.854 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:08:56.855 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:08:56.855 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:08:56.859 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:08:57.981 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:08:57.981 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:08:57.988 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367337 2023-10-03T21:08:57.989 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367337 -le 1696367755 ]] 2023-10-03T21:08:57.989 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:08:57.989 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:08:57.989 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:08:59.246 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:59.238+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:08:59.246 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:59.238+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:08:59.246 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:08:59.238+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:09:08.001 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:09:08.013 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:09:08.042 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:09:08.071 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:09:09.228 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:09:09.234 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:09:09.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367349 2023-10-03T21:09:09.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367349 -le 1696367755 ]] 2023-10-03T21:09:09.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:09:09.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:09:09.236 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:09:11.003 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:10.994+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:09:11.313 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:09:11.314 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:09:11.315 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:09:11.315 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:09:11.315 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:11.302+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:09:11.433 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:11.426+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:09:11.442 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:11.434+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:09:11.442 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:11.434+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:09:14.939 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:14.930+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:09:15.224 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:15.218+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:09:15.229 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:15.222+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:09:15.247 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:15.242+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:09:15.247 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:15.242+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:09:15.247 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:15.242+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:09:15.247 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:15.242+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:09:19.238 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:09:19.239 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:09:19.243 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:09:19.247 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:09:20.540 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:09:20.540 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:09:20.541 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367360 2023-10-03T21:09:20.541 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367360 -le 1696367755 ]] 2023-10-03T21:09:20.541 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:09:20.541 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:09:20.541 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:09:29.706 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:29.702+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:09:30.543 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:09:30.543 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:09:30.556 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:09:30.557 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:09:30.786 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:30.782+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:09:30.795 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:30.790+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:09:30.819 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:30.818+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:09:30.820 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:30.818+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:09:31.064 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:31.054+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:09:31.072 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:31.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:09:31.072 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:31.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:09:31.859 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:09:31.859 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:09:31.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367371 2023-10-03T21:09:31.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367371 -le 1696367755 ]] 2023-10-03T21:09:31.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:09:31.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:09:31.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:09:31.981 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:31.978+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:09:31.987 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:31.986+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:09:31.995 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:31.990+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:09:32.010 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:32.006+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:09:32.010 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:32.006+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:09:41.893 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:09:41.898 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:09:41.902 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:09:41.902 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:09:42.968 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:42.961+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:09:42.993 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:42.989+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:09:43.098 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:09:43.099 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:09:43.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367383 2023-10-03T21:09:43.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367383 -le 1696367755 ]] 2023-10-03T21:09:43.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:09:43.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:09:43.100 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:09:53.106 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:09:53.107 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:09:53.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:09:53.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:09:53.544 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:53.541+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:09:53.544 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:53.541+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:09:53.544 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:09:53.541+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:09:54.400 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:09:54.400 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:09:54.401 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367394 2023-10-03T21:09:54.401 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367394 -le 1696367755 ]] 2023-10-03T21:09:54.401 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:09:54.401 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:09:54.401 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:09:59.635 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:59.633+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:09:59.653 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:09:59.641+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:10:00.679 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:00.677+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:10:00.698 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:00.693+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:00.709 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:00.705+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:10:00.718 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:00.713+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:10:01.954 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:01.949+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:10:01.954 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:01.949+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:10:01.970 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:01.961+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:01.985 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:01.981+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:10:01.986 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:01.981+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:10:01.986 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:01.981+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:10:04.028 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:04.025+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:10:04.028 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:04.025+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:10:04.028 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:04.025+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:10:04.028 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:04.025+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:10:04.402 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:10:04.403 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:10:04.406 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:10:04.431 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:10:05.742 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:10:05.742 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:10:05.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367405 2023-10-03T21:10:05.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367405 -le 1696367755 ]] 2023-10-03T21:10:05.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:10:05.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:10:05.754 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:10:15.783 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:10:15.787 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:10:15.788 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:10:15.788 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:10:17.259 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:10:17.259 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:10:17.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367417 2023-10-03T21:10:17.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367417 -le 1696367755 ]] 2023-10-03T21:10:17.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:10:17.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:10:17.261 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:10:18.108 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:18.105+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:10:18.109 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:18.105+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:10:18.111 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:18.109+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:18.149 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:18.145+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:10:18.149 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:18.145+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:10:22.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:22.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:10:24.955 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:24.949+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:24.960 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:24.957+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:10:27.265 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:10:27.265 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:10:27.274 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:10:27.279 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:10:27.739 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:27.737+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:10:27.749 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:27.741+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:10:27.757 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:10:27.753+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:10:28.545 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:10:28.546 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:10:28.546 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367428 2023-10-03T21:10:28.546 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367428 -le 1696367755 ]] 2023-10-03T21:10:28.546 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:10:28.546 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:10:28.547 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:10:38.566 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:10:38.566 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:10:38.568 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:10:38.592 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:10:39.847 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:10:39.853 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:10:39.854 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367439 2023-10-03T21:10:39.854 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367439 -le 1696367755 ]] 2023-10-03T21:10:39.854 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:10:39.855 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:10:39.855 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:10:42.442 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:42.436+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:45.380 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.376+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:45.392 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.388+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:10:45.946 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.940+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:45.965 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.956+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:10:45.965 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.956+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:10:45.966 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.956+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:10:45.966 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.956+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:10:45.995 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:45.984+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:10:47.234 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:47.224+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:10:47.234 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:10:47.224+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:10:49.868 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:10:49.869 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:10:49.871 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:10:49.894 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:10:51.249 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:10:51.250 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:10:51.250 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367451 2023-10-03T21:10:51.250 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367451 -le 1696367755 ]] 2023-10-03T21:10:51.250 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:10:51.251 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:10:51.251 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:11:00.193 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:00.184+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:11:00.193 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:00.184+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:11:00.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:00.204+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:11:00.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:00.204+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:11:01.256 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:11:01.258 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:11:01.261 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:11:01.263 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:11:02.364 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:11:02.364 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:11:02.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367462 2023-10-03T21:11:02.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367462 -le 1696367755 ]] 2023-10-03T21:11:02.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:11:02.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:11:02.365 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:11:03.011 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:03.000+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:11:03.011 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:03.004+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:11:03.011 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:03.004+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:11:03.017 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:03.012+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:11:03.021 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:03.016+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:11:03.022 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:03.016+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:11:12.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:12.092+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:11:12.315 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:12.308+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:11:12.323 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:12.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:11:12.324 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:12.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:11:12.370 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:11:12.370 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:11:12.371 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:11:12.417 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:11:14.024 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:11:14.026 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:11:14.029 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367474 2023-10-03T21:11:14.029 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367474 -le 1696367755 ]] 2023-10-03T21:11:14.029 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:11:14.029 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:11:14.029 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:11:22.237 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:22.227+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:11:22.238 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:22.227+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:11:22.759 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:22.747+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:11:22.759 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:22.751+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:11:24.034 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:11:24.034 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:11:24.044 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:11:24.048 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367485 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367485 -le 1696367755 ]] 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:11:25.438 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:11:25.817 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:25.811+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:11:25.817 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:25.811+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:11:25.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:25.831+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:11:25.838 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:25.831+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:11:27.126 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:27.119+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:11:27.141 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:11:27.135+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:11:35.440 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:11:35.440 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:11:35.450 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:11:35.453 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:11:36.656 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:11:36.666 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:11:36.680 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367496 2023-10-03T21:11:36.680 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367496 -le 1696367755 ]] 2023-10-03T21:11:36.680 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:11:36.680 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:11:36.680 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:11:46.682 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:11:46.682 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:11:46.686 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:11:46.729 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:11:47.984 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:11:47.986 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:11:47.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367507 2023-10-03T21:11:47.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367507 -le 1696367755 ]] 2023-10-03T21:11:47.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:11:47.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:11:47.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:11:49.725 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:49.719+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:11:49.725 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:49.719+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:11:49.731 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:49.723+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:11:49.731 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:11:49.723+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:11:58.006 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:11:58.006 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:11:58.006 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:11:58.006 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:11:59.072 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:11:59.072 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:11:59.087 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367519 2023-10-03T21:11:59.087 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367519 -le 1696367755 ]] 2023-10-03T21:11:59.087 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:11:59.087 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:11:59.087 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:12:06.111 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:06.103+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:12:06.111 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:06.103+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:12:06.111 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:06.103+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:12:06.125 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:06.119+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:12:06.135 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:06.127+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:12:06.135 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:06.127+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:12:09.119 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:12:09.121 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:12:09.130 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:12:09.151 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:12:10.549 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:12:10.550 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:12:10.550 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367530 2023-10-03T21:12:10.550 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367530 -le 1696367755 ]] 2023-10-03T21:12:10.550 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:12:10.551 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:12:10.551 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:12:18.053 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.042+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:12:18.059 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.050+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:12:18.059 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.050+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:12:18.060 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.050+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:12:18.068 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:12:18.068 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:12:18.068 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:18.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:12:20.553 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:12:20.558 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:12:20.560 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:12:20.560 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:12:22.016 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:12:22.016 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:12:22.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367542 2023-10-03T21:12:22.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367542 -le 1696367755 ]] 2023-10-03T21:12:22.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:12:22.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:12:22.017 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:12:28.153 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:28.146+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:12:28.162 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:28.154+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:12:32.021 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:12:32.021 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:12:32.022 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:12:32.030 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:12:33.406 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:12:33.420 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:12:33.421 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367553 2023-10-03T21:12:33.421 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367553 -le 1696367755 ]] 2023-10-03T21:12:33.421 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:12:33.422 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:12:33.422 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:12:43.425 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:12:43.437 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:12:43.438 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:12:43.456 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:12:43.724 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.710+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:12:43.739 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.734+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:12:43.739 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.734+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:12:43.740 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.734+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:12:43.771 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.762+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:12:43.799 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.790+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:12:43.810 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.802+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:12:43.816 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.810+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:12:43.816 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:43.810+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:12:44.556 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:12:44.560 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:12:44.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367564 2023-10-03T21:12:44.561 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367564 -le 1696367755 ]] 2023-10-03T21:12:44.562 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:12:44.562 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:12:44.562 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:12:45.615 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.606+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:12:45.616 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.606+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:12:45.616 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.606+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:12:45.634 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.626+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:12:45.634 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.626+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:12:45.634 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.626+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:12:45.634 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:45.626+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:12:46.599 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:46.594+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:12:46.616 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:46.610+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:12:46.626 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:46.618+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:12:46.626 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:12:46.618+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:12:52.480 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:52.474+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:12:52.480 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:52.474+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:12:52.484 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:52.474+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:12:52.493 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:52.482+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:12:53.193 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:53.186+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:12:53.194 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:53.186+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:12:53.194 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:53.186+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:12:53.200 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:53.194+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:12:53.200 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:53.194+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:12:53.205 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:12:53.198+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:12:54.565 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:12:54.570 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:12:54.579 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:12:54.587 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:12:55.786 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:12:55.788 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:12:55.789 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367575 2023-10-03T21:12:55.789 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367575 -le 1696367755 ]] 2023-10-03T21:12:55.789 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:12:55.790 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:12:55.790 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:13:00.457 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:00.454+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:13:00.458 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:00.454+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:13:05.792 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:13:05.792 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:13:05.795 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:13:05.796 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:13:07.133 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:07.125+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:13:07.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:07.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:13:07.163 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:07.153+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:13:07.220 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:13:07.223 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:13:07.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367587 2023-10-03T21:13:07.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367587 -le 1696367755 ]] 2023-10-03T21:13:07.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:13:07.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:13:07.225 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:13:11.353 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:11.345+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:17.234 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:13:17.235 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:13:17.250 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:13:17.271 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:13:17.594 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:17.589+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:13:17.594 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:17.589+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:13:17.594 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:17.589+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:13:17.616 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:17.605+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:13:17.616 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:17.605+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:13:18.228 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.225+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:13:18.233 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.229+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:13:18.234 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.229+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:13:18.234 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.229+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:13:18.240 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.233+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:13:18.240 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.237+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:13:18.240 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:18.237+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:13:18.443 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:13:18.450 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:13:18.451 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367598 2023-10-03T21:13:18.451 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367598 -le 1696367755 ]] 2023-10-03T21:13:18.451 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:13:18.452 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:13:18.452 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:13:21.776 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:21.773+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:13:21.796 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:21.793+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:13:22.068 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:22.065+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:13:22.069 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:22.065+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:13:22.081 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:22.077+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:13:22.094 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:22.089+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:13:28.164 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:28.157+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:13:28.164 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:28.157+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:13:28.173 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:28.169+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:13:28.179 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:28.173+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:13:28.411 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:28.401+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:28.453 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:13:28.454 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:13:28.454 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:13:28.479 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:13:29.763 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:13:29.763 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:13:29.763 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367609 2023-10-03T21:13:29.763 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367609 -le 1696367755 ]] 2023-10-03T21:13:29.763 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:13:29.764 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:13:29.764 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:13:30.421 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.417+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:13:30.421 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.417+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:13:30.422 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.417+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:13:30.430 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.425+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:13:30.430 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.425+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:13:30.431 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.425+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:13:30.431 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.425+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:13:30.454 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.449+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:30.454 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.449+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:13:30.454 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.449+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:13:30.455 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.449+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:13:30.455 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.449+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:13:30.455 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:30.449+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:13:30.966 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:30.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:13:30.966 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:30.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:13:30.981 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:30.977+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:13:31.000 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:30.997+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:13:37.830 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:37.825+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:37.844 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:37.841+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:13:37.844 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:37.841+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:13:37.845 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:37.841+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:13:39.780 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:13:39.782 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:13:39.786 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:13:39.787 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:13:41.111 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:13:41.111 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:13:41.111 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367621 2023-10-03T21:13:41.111 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367621 -le 1696367755 ]] 2023-10-03T21:13:41.111 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:13:41.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:13:41.112 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:13:46.138 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:46.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:13:46.138 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:46.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:13:46.138 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:46.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:13:46.138 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:13:46.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:13:46.160 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:46.141+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:48.571 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:48.565+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:51.120 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:13:51.120 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:13:51.121 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:13:51.127 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:13:52.316 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:13:52.316 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:13:52.317 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367632 2023-10-03T21:13:52.317 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367632 -le 1696367755 ]] 2023-10-03T21:13:52.317 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:13:52.317 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:13:52.317 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:13:55.760 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:55.753+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:13:55.775 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:55.765+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:55.782 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:55.777+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:13:55.782 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:55.777+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:13:57.246 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:57.240+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:13:57.247 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:57.240+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:13:57.252 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:57.248+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:13:57.274 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:57.268+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:13:57.275 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:57.268+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:13:57.282 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:13:57.272+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:14:02.002 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:01.988+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:02.320 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:14:02.323 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:14:02.324 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:14:02.325 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:14:02.708 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:02.704+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:02.712 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:02.708+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:14:02.712 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:02.708+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:14:02.712 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:02.708+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:14:03.464 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:14:03.468 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:14:03.469 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367643 2023-10-03T21:14:03.470 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367643 -le 1696367755 ]] 2023-10-03T21:14:03.470 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:14:03.470 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:14:03.470 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:14:06.306 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:06.300+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:07.307 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:07.300+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:14:07.328 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:07.316+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:14:08.820 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:08.808+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:14:08.820 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:08.808+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:14:11.185 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:11.180+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:14:11.185 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:11.180+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:14:11.199 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:11.192+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:11.782 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:11.776+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:14:11.783 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:11.776+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:14:11.787 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:11.780+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:14:11.799 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:11.792+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:14:12.500 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:12.496+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:14:13.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:14:13.471 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:14:13.475 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:14:13.514 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:14:14.968 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:14:14.971 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:14:14.973 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367654 2023-10-03T21:14:14.973 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367654 -le 1696367755 ]] 2023-10-03T21:14:14.973 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:14:14.973 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:14:14.974 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:14:22.707 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.696+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:14:22.711 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.704+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:14:22.711 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.704+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:14:22.726 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.720+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:22.740 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.736+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:14:22.740 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.736+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:14:22.740 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.736+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:14:22.740 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.736+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:14:22.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:22.896+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:24.976 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:14:24.979 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:14:24.983 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:14:24.985 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:14:26.216 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:14:26.218 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:14:26.219 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367666 2023-10-03T21:14:26.219 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367666 -le 1696367755 ]] 2023-10-03T21:14:26.219 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:14:26.219 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:14:26.220 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:14:27.139 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:27.132+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:14:29.751 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:29.744+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:14:29.751 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:29.744+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:14:29.758 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:29.752+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:29.760 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:29.752+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:14:29.760 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:29.752+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:14:36.228 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:14:36.229 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:14:36.243 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:14:36.243 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:14:36.690 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.684+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:14:36.690 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.684+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:14:36.690 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.684+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:14:36.691 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.684+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:14:36.691 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.684+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:14:36.824 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:14:36.812+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:14:37.527 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:14:37.528 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:14:37.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367677 2023-10-03T21:14:37.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367677 -le 1696367755 ]] 2023-10-03T21:14:37.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:14:37.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:14:37.529 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:14:43.953 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.944+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:14:43.956 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.952+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:14:43.956 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.952+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:14:43.956 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.952+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:14:43.957 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.952+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:14:43.963 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.956+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:14:43.967 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.960+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:14:43.967 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.960+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:14:43.967 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.960+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:14:43.967 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.960+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:14:43.968 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:14:43.964+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:14:47.531 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:14:47.531 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:14:47.544 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:14:47.551 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:14:48.755 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:14:48.770 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:14:48.771 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367688 2023-10-03T21:14:48.771 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367688 -le 1696367755 ]] 2023-10-03T21:14:48.771 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:14:48.771 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:14:48.772 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:14:58.777 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:14:58.778 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:14:58.779 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:14:58.780 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:15:00.202 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:15:00.203 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:15:00.203 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367700 2023-10-03T21:15:00.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367700 -le 1696367755 ]] 2023-10-03T21:15:00.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:15:00.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:15:00.204 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:15:00.915 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:00.907+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:15:00.925 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:00.919+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:15:00.940 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:00.935+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:15:00.941 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:00.935+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:15:10.209 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:15:10.211 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:15:10.211 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:15:10.213 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:15:11.566 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:15:11.578 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:15:11.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367711 2023-10-03T21:15:11.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367711 -le 1696367755 ]] 2023-10-03T21:15:11.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:15:11.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:15:11.580 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:15:17.341 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:17.335+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:15:17.341 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:17.335+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:15:17.353 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:17.347+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:15:18.098 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.091+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:15:18.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.099+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:15:18.105 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.099+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:15:18.121 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.115+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:15:18.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.143+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:15:18.151 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.143+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:15:18.162 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.155+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:15:18.162 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:18.155+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:15:21.581 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:15:21.582 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:15:21.586 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:15:21.586 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:15:23.027 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:15:23.030 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:15:23.031 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367723 2023-10-03T21:15:23.032 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367723 -le 1696367755 ]] 2023-10-03T21:15:23.032 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:15:23.032 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:15:23.032 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:15:26.093 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:26.087+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:15:26.093 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:26.087+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:15:26.116 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:26.107+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:15:27.035 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:27.023+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:15:27.038 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:27.031+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:15:27.038 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:27.031+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:15:27.038 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:27.031+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:15:27.038 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:27.031+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:15:27.038 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:27.031+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:15:33.040 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:15:33.048 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:15:33.056 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:15:33.059 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:15:33.641 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:33.635+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:15:33.641 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:33.635+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:15:33.649 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:33.639+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:15:33.657 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:33.651+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:15:33.657 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:33.651+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:15:34.240 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:15:34.240 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:15:34.241 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367734 2023-10-03T21:15:34.241 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367734 -le 1696367755 ]] 2023-10-03T21:15:34.241 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:15:34.241 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:15:34.241 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:15:38.346 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:15:38.338+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:15:41.280 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.258+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:15:41.280 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.270+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:15:41.280 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.270+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:15:41.281 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.270+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:15:41.281 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.274+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:15:41.292 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.286+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:15:41.299 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:41.290+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:15:44.246 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:15:44.247 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:15:44.247 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:15:44.249 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:15:45.483 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:15:45.487 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:15:45.488 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367745 2023-10-03T21:15:45.489 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367745 -le 1696367755 ]] 2023-10-03T21:15:45.489 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ ! -z 172.21.15.167:0/2335679350 ]] 2023-10-03T21:15:45.489 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 172.21.15.167:0/2335679350 != \1\7\2\.\2\1\.\1\5\.\1\6\7\:\0\/\2\3\3\5\6\7\9\3\5\0 ]] 2023-10-03T21:15:45.489 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:15:55.495 INFO:tasks.workunit.client.0.smithi167.stderr:++ ceph mgr dump 2023-10-03T21:15:55.496 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T21:15:55.499 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T21:15:55.504 INFO:tasks.workunit.client.0.smithi167.stderr:++ jq '.active_clients[]' 2023-10-03T21:15:56.927 INFO:tasks.workunit.client.0.smithi167.stderr:+ CLIENT_ADDR=172.21.15.167:0/2335679350 2023-10-03T21:15:56.927 INFO:tasks.workunit.client.0.smithi167.stderr:++ date +%s 2023-10-03T21:15:56.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ CURRENT_TIME=1696367756 2023-10-03T21:15:56.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ [[ 1696367756 -le 1696367755 ]] 2023-10-03T21:15:56.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i = 1 )) 2023-10-03T21:15:56.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:15:56.929 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:15:57.189 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:15:57.182+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:15:57.189 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:15:57.196 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:15:58.066 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:58.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:15:58.066 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:58.058+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:15:58.077 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:58.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:15:58.086 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:58.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:15:58.086 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:15:58.078+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:05.132 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:05.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:16:05.137 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:05.130+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:16:05.137 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:05.130+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:16:05.144 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:05.138+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:05.163 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:05.150+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:16:05.164 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:05.150+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:07.197 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:16:07.197 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:16:07.198 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:16:07.309 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:16:07.302+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:16:07.310 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:16:07.351 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:16:08.418 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.410+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:16:08.420 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.414+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:16:08.421 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.414+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:16:08.442 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.434+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:16:08.443 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.434+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:16:08.443 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.434+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:16:08.443 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.434+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:16:08.455 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.434+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:16:08.456 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.438+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:16:08.462 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.438+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:16:08.462 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.438+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:16:08.462 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.438+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:16:08.463 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:08.438+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:16:12.413 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:12.402+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:12.421 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:12.414+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:14.301 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:14.294+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:16:14.310 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:14.302+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:14.329 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:14.314+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:15.262 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:15.254+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:16:15.278 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:15.270+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:16:15.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:15.270+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:16:15.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:15.270+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:16:17.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:16:17.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:16:17.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:16:17.455 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:16:17.446+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:16:17.459 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:16:17.482 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:16:18.465 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:18.462+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:18.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:18.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:16:18.478 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:18.474+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:27.507 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:16:27.507 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:16:27.507 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:16:27.689 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:16:27.686+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:16:27.689 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:16:27.699 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:16:34.554 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:34.549+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:16:34.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:34.561+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:34.572 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:34.569+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:16:34.574 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:34.569+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:36.473 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:36.469+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:16:37.701 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:16:37.701 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:16:37.701 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:16:37.830 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:16:37.825+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:16:37.831 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:16:37.860 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:16:39.464 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:39.457+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:16:39.478 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:39.473+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:16:39.494 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:39.489+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:16:39.494 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:39.489+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:16:39.512 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:39.509+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:16:43.092 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.077+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:16:43.092 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:16:43.094 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:16:43.094 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.085+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:16:43.101 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.093+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:43.101 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.093+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:16:43.101 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:43.093+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:46.216 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:46.209+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:16:46.216 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:46.209+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:16:46.220 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:46.217+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:16:46.228 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:46.221+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:16:46.228 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:46.221+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:16:46.243 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:46.237+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:16:46.243 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:46.237+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:16:46.250 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:46.245+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:16:46.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:46.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:16:46.255 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:16:46.249+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:16:47.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:16:47.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:16:47.861 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:16:47.979 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:16:47.973+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:16:47.979 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:16:47.990 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:16:50.891 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:50.881+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:16:50.891 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:50.881+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:16:50.905 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:50.893+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:16:50.922 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:50.909+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:16:57.991 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:16:57.992 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:16:57.992 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:16:58.138 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:16:58.133+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:16:58.139 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:16:58.150 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:16:59.826 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:16:59.821+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:03.092 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.085+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:17:03.107 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.101+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:17:03.108 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.101+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:17:03.126 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:17:03.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:03.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:17:03.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:17:03.127 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:17:03.128 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:17:03.128 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:03.121+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:03.656 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:03.649+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:17:03.664 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:03.661+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:17:03.664 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:03.661+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:17:03.668 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:03.665+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:17:03.677 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:03.673+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:17:03.680 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:03.673+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:17:04.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:04.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:17:04.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:04.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:17:04.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:04.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:17:04.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:04.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:17:08.151 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:17:08.152 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:17:08.152 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:17:08.253 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:17:08.249+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:17:08.253 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:17:08.273 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:17:09.803 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:09.797+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:17:09.804 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:09.797+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:17:09.804 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:09.797+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:17:09.807 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:09.805+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:17:09.820 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:09.813+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:17:12.703 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:12.693+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:17:12.715 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:12.709+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:12.728 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:12.721+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:17:12.741 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:12.737+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:17:12.741 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:12.737+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:17:12.748 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:12.741+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:17.902 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:17.889+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:17.925 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:17.921+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:18.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:17:18.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:17:18.275 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:17:18.419 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:17:18.413+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:17:18.424 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:17:18.432 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:17:28.434 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:17:28.434 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:17:28.434 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:17:28.652 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:17:28.644+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:17:28.657 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:17:28.702 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:17:30.862 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:30.856+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:17:30.873 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:30.868+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:17:31.842 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:31.832+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:34.056 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:34.048+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:17:38.708 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:17:38.708 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:17:38.708 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:17:38.806 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:17:38.800+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:17:38.806 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:17:38.815 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:17:42.435 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.424+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:17:42.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.440+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:17:42.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.444+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:17:42.448 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.444+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:17:42.456 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.448+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:17:42.456 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.448+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:17:42.456 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:17:42.448+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:17:43.379 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:43.372+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:17:43.389 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:43.384+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:43.391 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:43.384+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:17:43.400 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:43.396+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:17:43.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:43.412+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:17:43.421 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:43.416+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:46.540 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:46.528+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:48.817 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:17:48.817 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:17:48.817 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:17:48.926 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:17:48.920+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:17:48.926 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:17:48.936 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:17:49.752 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:49.744+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:49.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:49.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:17:49.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:49.768+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:17:49.773 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:49.768+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:52.028 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:52.016+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:17:52.040 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:17:52.032+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:17:58.942 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:17:58.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:17:58.943 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:17:59.074 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:17:59.068+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:17:59.075 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:17:59.113 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:18:03.029 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:03.016+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:18:03.030 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:03.016+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:18:03.039 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:03.028+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:18:03.042 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:03.036+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:18:09.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:18:09.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:18:09.128 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:18:09.159 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:09.152+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:18:09.191 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:09.184+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:18:09.282 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:18:09.276+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:18:09.283 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:18:09.303 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:18:10.368 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:10.364+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:18:12.165 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:12.160+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:18:12.165 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:12.160+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:18:19.307 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:18:19.307 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:18:19.307 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:18:19.382 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:18:19.375+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:18:19.383 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:18:19.387 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:18:22.301 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.295+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:18:22.309 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.303+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:18:22.327 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.319+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:18:22.342 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.335+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:18:22.344 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.339+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:18:22.344 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.339+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:18:22.344 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:22.339+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:18:29.389 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:18:29.389 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:18:29.389 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:18:29.550 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:18:29.543+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:18:29.550 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:18:29.566 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:18:29.576 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:29.567+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:18:29.576 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:29.567+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:18:29.576 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:29.567+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:18:29.594 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:18:29.587+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:18:32.503 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.495+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:18:32.530 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.519+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:18:32.531 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.523+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:18:32.541 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.535+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:18:32.541 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.535+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:18:32.541 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.535+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:18:32.542 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:32.535+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:18:39.567 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:18:39.568 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:18:39.568 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:18:39.691 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:18:39.683+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:18:39.692 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:18:39.696 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:18:45.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:45.107+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:18:45.115 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:45.107+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:18:49.730 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:18:49.730 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:18:49.730 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:18:49.845 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:18:49.839+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:18:49.845 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:18:49.849 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:18:52.408 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:52.391+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:18:52.428 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:52.423+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:18:52.428 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:52.423+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:18:52.429 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:52.423+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:18:52.429 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:52.423+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:18:55.143 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:55.135+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:18:59.591 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:18:59.583+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:18:59.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:18:59.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:18:59.851 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:18:59.937 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:18:59.931+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:18:59.937 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:18:59.979 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:19:05.927 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:05.919+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:19:05.927 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:05.919+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:19:05.927 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:05.919+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:19:09.981 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:19:09.981 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:19:09.981 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:19:10.173 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:19:10.163+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:19:10.175 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:19:10.192 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:19:13.209 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:13.203+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:19:13.209 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:13.203+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:19:13.209 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:13.203+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:19:17.410 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:17.402+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:19:17.410 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:17.402+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:19:17.410 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:17.402+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:19:20.198 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:19:20.198 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:19:20.198 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:19:20.332 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:19:20.326+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:19:20.336 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:19:20.355 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:19:21.990 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:21.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:19:21.997 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:21.990+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:19:30.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:19:30.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:19:30.357 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:19:30.492 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:19:30.482+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:19:30.492 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:19:30.512 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:19:32.155 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.142+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:19:32.174 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.162+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:19:32.174 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.162+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:19:32.174 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.162+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:19:32.228 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.218+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:19:32.228 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.218+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:19:32.228 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:32.218+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:19:35.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:35.750+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:19:35.759 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:35.750+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:19:35.764 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:35.758+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:19:35.772 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:35.762+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:19:40.514 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:19:40.515 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:19:40.515 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:19:40.624 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:19:40.618+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:19:40.625 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:19:40.629 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:19:44.352 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:44.342+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:19:46.187 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:46.178+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:19:46.209 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:46.202+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:19:46.212 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:46.206+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:19:46.226 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:46.218+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:19:50.630 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:19:50.630 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:19:50.631 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-03T21:19:50.735 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:19:50.726+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:19:50.735 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:19:50.768 INFO:tasks.workunit.client.0.smithi167.stderr:+ sleep 10 2023-10-03T21:19:53.081 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:53.062+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:19:53.082 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:53.062+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:19:53.082 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:53.070+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:19:53.083 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:19:53.074+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:19:54.279 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:19:54.270+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:20:00.534 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.530+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:20:00.534 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.530+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:20:00.534 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.530+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:20:00.535 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.530+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:20:00.567 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:20:00.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:20:00.568 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:00.562+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:20:00.778 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i++ )) 2023-10-03T21:20:00.778 INFO:tasks.workunit.client.0.smithi167.stderr:+ (( i <= 24 )) 2023-10-03T21:20:00.781 INFO:tasks.workunit.client.0.smithi167.stderr:+ grep 'every 2m' 2023-10-03T21:20:00.792 INFO:tasks.workunit.client.0.smithi167.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-03T21:20:00.944 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-03T21:20:00.942+0000 7f6ec8296640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-03T21:20:00.944 INFO:tasks.workunit.client.0.smithi167.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-03T21:20:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T21:20:01.057 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-03T21:20:01.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-03T21:20:01.845 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T21:20:01.917 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=076c84df32724e61ae8155dfd15f1868 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T21:20:01.919 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-03T21:20:01.967 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-03T21:20:01.977 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-03T21:20:01.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:20:02.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.646+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:20:02.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.646+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:20:02.651 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.646+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:20:02.666 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.662+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:20:02.666 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.662+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:20:02.666 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.662+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:20:02.666 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.662+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:20:02.666 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:02.662+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:20:02.732 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:02.733 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:20:02.805 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2088,"stamp":"2023-10-03T21:20:02.087898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39780531730,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1740910,"num_read_kb":6699825,"num_write":1696615,"num_write_kb":65852444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64658,"num_flush":14746,"num_flush_kb":59290185,"num_evict":9525,"num_evict_kb":38374069,"num_promote":9563,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120681,"ondisk_log_size":120681,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":44917616,"kb_used_data":41235544,"kb_used_omap":1935,"kb_used_meta":3680112,"kb_avail":238197904,"statfs":{"total":289910292480,"available":243914653696,"internally_reserved":0,"allocated":42225197056,"data_stored":48299041040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":3768435555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,9,2,4,9,15,20,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32440352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4333,"num_read_kb":17184,"num_write":4316,"num_write_kb":188685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":32,"num_flush":42,"num_flush_kb":171624,"num_evict":16,"num_evict_kb":65316,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006623"},"pg_stats":[{"pgid":"2.7","version":"89'200571","reported_seq":402244,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.547122+0000","last_change":"2023-10-03T20:19:04.303179+0000","last_active":"2023-10-03T21:19:57.547122+0000","last_peered":"2023-10-03T21:19:57.547122+0000","last_clean":"2023-10-03T21:19:57.547122+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:19:57.547122+0000","last_undegraded":"2023-10-03T21:19:57.547122+0000","last_fullsized":"2023-10-03T21:19:57.547122+0000","mapping_epoch":12,"log_start":"85'190513","ondisk_log_start":"85'190513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:29:24.435845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4872790350,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":201516,"num_read_kb":786978,"num_write":199412,"num_write_kb":9430998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'200739","reported_seq":404659,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.529143+0000","last_change":"2023-10-03T20:19:04.232076+0000","last_active":"2023-10-03T21:19:57.529143+0000","last_peered":"2023-10-03T21:19:57.529143+0000","last_clean":"2023-10-03T21:19:57.529143+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:19:57.529143+0000","last_undegraded":"2023-10-03T21:19:57.529143+0000","last_fullsized":"2023-10-03T21:19:57.529143+0000","mapping_epoch":12,"log_start":"85'190713","ondisk_log_start":"85'190713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:27:33.334820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4933607743,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":204127,"num_read_kb":793580,"num_write":199563,"num_write_kb":8968308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'218056","reported_seq":437910,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.519475+0000","last_change":"2023-10-03T20:19:04.308490+0000","last_active":"2023-10-03T21:19:57.519475+0000","last_peered":"2023-10-03T21:19:57.519475+0000","last_clean":"2023-10-03T21:19:57.519475+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:19:57.519475+0000","last_undegraded":"2023-10-03T21:19:57.519475+0000","last_fullsized":"2023-10-03T21:19:57.519475+0000","mapping_epoch":12,"log_start":"86'208008","ondisk_log_start":"86'208008","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:03.359890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5152940535,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":219741,"num_read_kb":859313,"num_write":216810,"num_write_kb":8723445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'207798","reported_seq":418542,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.518324+0000","last_change":"2023-10-03T20:19:04.206725+0000","last_active":"2023-10-03T21:19:57.518324+0000","last_peered":"2023-10-03T21:19:57.518324+0000","last_clean":"2023-10-03T21:19:57.518324+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:19:57.518324+0000","last_undegraded":"2023-10-03T21:19:57.518324+0000","last_fullsized":"2023-10-03T21:19:57.518324+0000","mapping_epoch":12,"log_start":"86'197714","ondisk_log_start":"86'197714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:08:51.601852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4908630499,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":210550,"num_read_kb":820754,"num_write":206582,"num_write_kb":7735409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'32002","reported_seq":52732,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:58.499001+0000","last_change":"2023-10-03T20:19:04.199607+0000","last_active":"2023-10-03T21:19:58.499001+0000","last_peered":"2023-10-03T21:19:58.499001+0000","last_clean":"2023-10-03T21:19:58.499001+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:19:58.499001+0000","last_undegraded":"2023-10-03T21:19:58.499001+0000","last_fullsized":"2023-10-03T21:19:58.499001+0000","mapping_epoch":15,"log_start":"84'21925","ondisk_log_start":"84'21925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:11:58.561538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163077789,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":17997,"num_read_kb":51699,"num_write":14766,"num_write_kb":49144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15963,"num_flush":4313,"num_flush_kb":17384936,"num_evict":2351,"num_evict_kb":9492321,"num_promote":2360,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'200321","reported_seq":402199,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:58.936361+0000","last_change":"2023-10-03T20:19:04.198050+0000","last_active":"2023-10-03T21:19:58.936361+0000","last_peered":"2023-10-03T21:19:58.936361+0000","last_clean":"2023-10-03T21:19:58.936361+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:19:58.936361+0000","last_undegraded":"2023-10-03T21:19:58.936361+0000","last_fullsized":"2023-10-03T21:19:58.936361+0000","mapping_epoch":12,"log_start":"86'190309","ondisk_log_start":"86'190309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:46:58.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4663857473,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":201723,"num_read_kb":786705,"num_write":199178,"num_write_kb":6836240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'31866","reported_seq":51960,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:56.372777+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:19:56.372777+0000","last_peered":"2023-10-03T21:19:56.372777+0000","last_clean":"2023-10-03T21:19:56.372777+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:19:56.372777+0000","last_undegraded":"2023-10-03T21:19:56.372777+0000","last_fullsized":"2023-10-03T21:19:56.372777+0000","mapping_epoch":15,"log_start":"84'21830","ondisk_log_start":"84'21830","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:22:56.156359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146472944,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":18894,"num_read_kb":49466,"num_write":13777,"num_write_kb":44771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16167,"num_flush":3239,"num_flush_kb":12994128,"num_evict":2436,"num_evict_kb":9785416,"num_promote":2447,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"89'210225","reported_seq":422699,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:59.559319+0000","last_change":"2023-10-03T20:19:04.230032+0000","last_active":"2023-10-03T21:19:59.559319+0000","last_peered":"2023-10-03T21:19:59.559319+0000","last_clean":"2023-10-03T21:19:59.559319+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:19:59.559319+0000","last_undegraded":"2023-10-03T21:19:59.559319+0000","last_fullsized":"2023-10-03T21:19:59.559319+0000","mapping_epoch":12,"log_start":"86'200213","ondisk_log_start":"86'200213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:04:12.017189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5035999653,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":212315,"num_read_kb":829586,"num_write":209006,"num_write_kb":7576815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'28114","reported_seq":46819,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:58.754398+0000","last_change":"2023-10-03T20:19:04.198355+0000","last_active":"2023-10-03T21:19:58.754398+0000","last_peered":"2023-10-03T21:19:58.754398+0000","last_clean":"2023-10-03T21:19:58.754398+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:19:58.754398+0000","last_undegraded":"2023-10-03T21:19:58.754398+0000","last_fullsized":"2023-10-03T21:19:58.754398+0000","mapping_epoch":15,"log_start":"84'18030","ondisk_log_start":"84'18030","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:29:49.227097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146473615,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16009,"num_read_kb":49516,"num_write":14121,"num_write_kb":46638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16788,"num_flush":3633,"num_flush_kb":14645356,"num_evict":2382,"num_evict_kb":9634261,"num_promote":2391,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"89'202971","reported_seq":408695,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:59.568412+0000","last_change":"2023-10-03T20:19:04.276516+0000","last_active":"2023-10-03T21:19:59.568412+0000","last_peered":"2023-10-03T21:19:59.568412+0000","last_clean":"2023-10-03T21:19:59.568412+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:19:59.568412+0000","last_undegraded":"2023-10-03T21:19:59.568412+0000","last_fullsized":"2023-10-03T21:19:59.568412+0000","mapping_epoch":12,"log_start":"86'192913","ondisk_log_start":"86'192913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:13:18.149220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4738806211,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":205525,"num_read_kb":802550,"num_write":201807,"num_write_kb":6857749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'39637","reported_seq":64353,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.326316+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:19:57.326316+0000","last_peered":"2023-10-03T21:19:57.326316+0000","last_clean":"2023-10-03T21:19:57.326316+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:19:57.326316+0000","last_undegraded":"2023-10-03T21:19:57.326316+0000","last_fullsized":"2023-10-03T21:19:57.326316+0000","mapping_epoch":15,"log_start":"84'29625","ondisk_log_start":"84'29625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:26:51.397031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150638150,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24026,"num_read_kb":55161,"num_write":14060,"num_write_kb":46014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15740,"num_flush":3561,"num_flush_kb":14265765,"num_evict":2356,"num_evict_kb":9462071,"num_promote":2365,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'208584","reported_seq":417161,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.506751+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:19:57.506751+0000","last_peered":"2023-10-03T21:19:57.506751+0000","last_clean":"2023-10-03T21:19:57.506751+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:19:57.506751+0000","last_undegraded":"2023-10-03T21:19:57.506751+0000","last_fullsized":"2023-10-03T21:19:57.506751+0000","mapping_epoch":12,"log_start":"86'198507","ondisk_log_start":"86'198507","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:28.300668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4866646400,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":208381,"num_read_kb":814426,"num_write":207396,"num_write_kb":9535019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":245,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T21:18:51.630010+0000","last_change":"2023-10-03T20:13:22.539837+0000","last_active":"2023-10-03T21:18:51.630010+0000","last_peered":"2023-10-03T21:18:51.630010+0000","last_clean":"2023-10-03T21:18:51.630010+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:18:51.630010+0000","last_undegraded":"2023-10-03T21:18:51.630010+0000","last_fullsized":"2023-10-03T21:18:51.630010+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:15:59.949043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606662498,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":76926,"num_read_kb":205842,"num_write":56724,"num_write_kb":186567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64658,"num_flush":14746,"num_flush_kb":59290185,"num_evict":9525,"num_evict_kb":38374069,"num_promote":9563,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1213599744,"data_stored":1213312738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39173278864,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1663878,"num_read_kb":6493892,"num_write":1639754,"num_write_kb":65663983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41012203520,"data_stored":47087053088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80375,"ondisk_log_size":80375,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8954904,"kb_used_data":8063056,"kb_used_omap":376,"kb_used_meta":891463,"kb_avail":85416936,"statfs":{"total":96636764160,"available":87466942464,"internally_reserved":0,"allocated":8256569344,"data_stored":9506944514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":912859037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.98299999999999998,"15min":1.097},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.1589999999999998,"5min":7.907,"15min":19.114000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0349999999999999,"15min":0.98799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":6.1660000000000004,"5min":10.976000000000001,"15min":15.244},"last":0.42399999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.9749999999999996,"5min":30.677,"15min":30.677},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.0179999999999998,"5min":1.766,"15min":1.423},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.285,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21808024,"kb_used_data":20313168,"kb_used_omap":961,"kb_used_meta":1493886,"kb_avail":72563816,"statfs":{"total":96636764160,"available":74305347584,"internally_reserved":0,"allocated":20800684032,"data_stored":23838092928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1529739301},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,8,14,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":2.8740000000000001,"5min":2.798,"15min":2.7360000000000002},"min":{"1min":0.22600000000000001,"5min":0.153,"15min":0.13800000000000001},"max":{"1min":15.569000000000001,"5min":28.088999999999999,"15min":28.088999999999999},"last":0.214},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.84799999999999998,"15min":1.369},"min":{"1min":0.189,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.4000000000000004,"5min":5.9660000000000002,"15min":14.116},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.1679999999999999,"15min":1.232},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":5.2850000000000001,"5min":11.577,"15min":24.614999999999998},"last":0.245},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.161,"15min":1.1619999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":1.417,"5min":6.7190000000000003,"15min":12.989000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359739170,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14154688,"kb_used_data":12859320,"kb_used_omap":597,"kb_used_meta":1294762,"kb_avail":80217152,"statfs":{"total":96636764160,"available":82142363648,"internally_reserved":0,"allocated":13167943680,"data_stored":14954003598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1325837217},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.212,"15min":1.3400000000000001},"min":{"1min":0.19800000000000001,"5min":0.16200000000000001,"15min":0.13900000000000001},"max":{"1min":6.601,"5min":9.8450000000000006,"15min":21.391999999999999},"last":4.9050000000000002},{"interface":"front","average":{"1min":1.379,"5min":1.2410000000000001,"15min":1.0229999999999999},"min":{"1min":0.23400000000000001,"5min":0.20200000000000001,"15min":0.182},"max":{"1min":11.045999999999999,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.38600000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0580000000000001,"15min":1.123},"min":{"1min":0.222,"5min":0.187,"15min":0.187},"max":{"1min":8.0410000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.97099999999999997,"15min":0.751},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":4.5819999999999999,"5min":6.827,"15min":8.077},"last":0.34999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12857470976,"data_stored":14643922856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20505448448,"data_stored":23542987920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7649284096,"data_stored":8900142312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309616640,"data_stored":309547308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297181184,"data_stored":297106998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606801920,"data_stored":606658432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:20:02.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:20:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:03.682 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:20:03.762 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2088,"stamp":"2023-10-03T21:20:02.087898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39780531730,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1740910,"num_read_kb":6699825,"num_write":1696615,"num_write_kb":65852444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64658,"num_flush":14746,"num_flush_kb":59290185,"num_evict":9525,"num_evict_kb":38374069,"num_promote":9563,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120681,"ondisk_log_size":120681,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":44917616,"kb_used_data":41235544,"kb_used_omap":1935,"kb_used_meta":3680112,"kb_avail":238197904,"statfs":{"total":289910292480,"available":243914653696,"internally_reserved":0,"allocated":42225197056,"data_stored":48299041040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":3768435555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,9,2,4,9,15,20,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32440352,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4333,"num_read_kb":17184,"num_write":4316,"num_write_kb":188685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":32,"num_flush":42,"num_flush_kb":171624,"num_evict":16,"num_evict_kb":65316,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006623"},"pg_stats":[{"pgid":"2.7","version":"89'200571","reported_seq":402244,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.547122+0000","last_change":"2023-10-03T20:19:04.303179+0000","last_active":"2023-10-03T21:19:57.547122+0000","last_peered":"2023-10-03T21:19:57.547122+0000","last_clean":"2023-10-03T21:19:57.547122+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:19:57.547122+0000","last_undegraded":"2023-10-03T21:19:57.547122+0000","last_fullsized":"2023-10-03T21:19:57.547122+0000","mapping_epoch":12,"log_start":"85'190513","ondisk_log_start":"85'190513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:29:24.435845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4872790350,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":201516,"num_read_kb":786978,"num_write":199412,"num_write_kb":9430998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'200739","reported_seq":404659,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.529143+0000","last_change":"2023-10-03T20:19:04.232076+0000","last_active":"2023-10-03T21:19:57.529143+0000","last_peered":"2023-10-03T21:19:57.529143+0000","last_clean":"2023-10-03T21:19:57.529143+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:19:57.529143+0000","last_undegraded":"2023-10-03T21:19:57.529143+0000","last_fullsized":"2023-10-03T21:19:57.529143+0000","mapping_epoch":12,"log_start":"85'190713","ondisk_log_start":"85'190713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:27:33.334820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4933607743,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":204127,"num_read_kb":793580,"num_write":199563,"num_write_kb":8968308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'218056","reported_seq":437910,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.519475+0000","last_change":"2023-10-03T20:19:04.308490+0000","last_active":"2023-10-03T21:19:57.519475+0000","last_peered":"2023-10-03T21:19:57.519475+0000","last_clean":"2023-10-03T21:19:57.519475+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:19:57.519475+0000","last_undegraded":"2023-10-03T21:19:57.519475+0000","last_fullsized":"2023-10-03T21:19:57.519475+0000","mapping_epoch":12,"log_start":"86'208008","ondisk_log_start":"86'208008","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:03.359890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5152940535,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":219741,"num_read_kb":859313,"num_write":216810,"num_write_kb":8723445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'207798","reported_seq":418542,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.518324+0000","last_change":"2023-10-03T20:19:04.206725+0000","last_active":"2023-10-03T21:19:57.518324+0000","last_peered":"2023-10-03T21:19:57.518324+0000","last_clean":"2023-10-03T21:19:57.518324+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:19:57.518324+0000","last_undegraded":"2023-10-03T21:19:57.518324+0000","last_fullsized":"2023-10-03T21:19:57.518324+0000","mapping_epoch":12,"log_start":"86'197714","ondisk_log_start":"86'197714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:08:51.601852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4908630499,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":210550,"num_read_kb":820754,"num_write":206582,"num_write_kb":7735409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'32002","reported_seq":52732,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:58.499001+0000","last_change":"2023-10-03T20:19:04.199607+0000","last_active":"2023-10-03T21:19:58.499001+0000","last_peered":"2023-10-03T21:19:58.499001+0000","last_clean":"2023-10-03T21:19:58.499001+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:19:58.499001+0000","last_undegraded":"2023-10-03T21:19:58.499001+0000","last_fullsized":"2023-10-03T21:19:58.499001+0000","mapping_epoch":15,"log_start":"84'21925","ondisk_log_start":"84'21925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:11:58.561538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163077789,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":17997,"num_read_kb":51699,"num_write":14766,"num_write_kb":49144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15963,"num_flush":4313,"num_flush_kb":17384936,"num_evict":2351,"num_evict_kb":9492321,"num_promote":2360,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'200321","reported_seq":402199,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:58.936361+0000","last_change":"2023-10-03T20:19:04.198050+0000","last_active":"2023-10-03T21:19:58.936361+0000","last_peered":"2023-10-03T21:19:58.936361+0000","last_clean":"2023-10-03T21:19:58.936361+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:19:58.936361+0000","last_undegraded":"2023-10-03T21:19:58.936361+0000","last_fullsized":"2023-10-03T21:19:58.936361+0000","mapping_epoch":12,"log_start":"86'190309","ondisk_log_start":"86'190309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:46:58.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4663857473,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":201723,"num_read_kb":786705,"num_write":199178,"num_write_kb":6836240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'31866","reported_seq":51960,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:56.372777+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:19:56.372777+0000","last_peered":"2023-10-03T21:19:56.372777+0000","last_clean":"2023-10-03T21:19:56.372777+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:19:56.372777+0000","last_undegraded":"2023-10-03T21:19:56.372777+0000","last_fullsized":"2023-10-03T21:19:56.372777+0000","mapping_epoch":15,"log_start":"84'21830","ondisk_log_start":"84'21830","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:22:56.156359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146472944,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":18894,"num_read_kb":49466,"num_write":13777,"num_write_kb":44771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16167,"num_flush":3239,"num_flush_kb":12994128,"num_evict":2436,"num_evict_kb":9785416,"num_promote":2447,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"89'210225","reported_seq":422699,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:59.559319+0000","last_change":"2023-10-03T20:19:04.230032+0000","last_active":"2023-10-03T21:19:59.559319+0000","last_peered":"2023-10-03T21:19:59.559319+0000","last_clean":"2023-10-03T21:19:59.559319+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:19:59.559319+0000","last_undegraded":"2023-10-03T21:19:59.559319+0000","last_fullsized":"2023-10-03T21:19:59.559319+0000","mapping_epoch":12,"log_start":"86'200213","ondisk_log_start":"86'200213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:04:12.017189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5035999653,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":212315,"num_read_kb":829586,"num_write":209006,"num_write_kb":7576815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'28114","reported_seq":46819,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:58.754398+0000","last_change":"2023-10-03T20:19:04.198355+0000","last_active":"2023-10-03T21:19:58.754398+0000","last_peered":"2023-10-03T21:19:58.754398+0000","last_clean":"2023-10-03T21:19:58.754398+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:19:58.754398+0000","last_undegraded":"2023-10-03T21:19:58.754398+0000","last_fullsized":"2023-10-03T21:19:58.754398+0000","mapping_epoch":15,"log_start":"84'18030","ondisk_log_start":"84'18030","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:29:49.227097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146473615,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16009,"num_read_kb":49516,"num_write":14121,"num_write_kb":46638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16788,"num_flush":3633,"num_flush_kb":14645356,"num_evict":2382,"num_evict_kb":9634261,"num_promote":2391,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"89'202971","reported_seq":408695,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:59.568412+0000","last_change":"2023-10-03T20:19:04.276516+0000","last_active":"2023-10-03T21:19:59.568412+0000","last_peered":"2023-10-03T21:19:59.568412+0000","last_clean":"2023-10-03T21:19:59.568412+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:19:59.568412+0000","last_undegraded":"2023-10-03T21:19:59.568412+0000","last_fullsized":"2023-10-03T21:19:59.568412+0000","mapping_epoch":12,"log_start":"86'192913","ondisk_log_start":"86'192913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:13:18.149220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4738806211,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":205525,"num_read_kb":802550,"num_write":201807,"num_write_kb":6857749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'39637","reported_seq":64353,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.326316+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:19:57.326316+0000","last_peered":"2023-10-03T21:19:57.326316+0000","last_clean":"2023-10-03T21:19:57.326316+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:19:57.326316+0000","last_undegraded":"2023-10-03T21:19:57.326316+0000","last_fullsized":"2023-10-03T21:19:57.326316+0000","mapping_epoch":15,"log_start":"84'29625","ondisk_log_start":"84'29625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:26:51.397031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150638150,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24026,"num_read_kb":55161,"num_write":14060,"num_write_kb":46014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15740,"num_flush":3561,"num_flush_kb":14265765,"num_evict":2356,"num_evict_kb":9462071,"num_promote":2365,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'208584","reported_seq":417161,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:57.506751+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:19:57.506751+0000","last_peered":"2023-10-03T21:19:57.506751+0000","last_clean":"2023-10-03T21:19:57.506751+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:19:57.506751+0000","last_undegraded":"2023-10-03T21:19:57.506751+0000","last_fullsized":"2023-10-03T21:19:57.506751+0000","mapping_epoch":12,"log_start":"86'198507","ondisk_log_start":"86'198507","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:28.300668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4866646400,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":208381,"num_read_kb":814426,"num_write":207396,"num_write_kb":9535019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":245,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T21:18:51.630010+0000","last_change":"2023-10-03T20:13:22.539837+0000","last_active":"2023-10-03T21:18:51.630010+0000","last_peered":"2023-10-03T21:18:51.630010+0000","last_clean":"2023-10-03T21:18:51.630010+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:18:51.630010+0000","last_undegraded":"2023-10-03T21:18:51.630010+0000","last_fullsized":"2023-10-03T21:18:51.630010+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:15:59.949043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606662498,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":76926,"num_read_kb":205842,"num_write":56724,"num_write_kb":186567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64658,"num_flush":14746,"num_flush_kb":59290185,"num_evict":9525,"num_evict_kb":38374069,"num_promote":9563,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1213599744,"data_stored":1213312738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39173278864,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1663878,"num_read_kb":6493892,"num_write":1639754,"num_write_kb":65663983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41012203520,"data_stored":47087053088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80375,"ondisk_log_size":80375,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8954904,"kb_used_data":8063056,"kb_used_omap":376,"kb_used_meta":891463,"kb_avail":85416936,"statfs":{"total":96636764160,"available":87466942464,"internally_reserved":0,"allocated":8256569344,"data_stored":9506944514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":912859037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.98299999999999998,"15min":1.097},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.1589999999999998,"5min":7.907,"15min":19.114000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0349999999999999,"15min":0.98799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":6.1660000000000004,"5min":10.976000000000001,"15min":15.244},"last":0.42399999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.9749999999999996,"5min":30.677,"15min":30.677},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.0179999999999998,"5min":1.766,"15min":1.423},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.285,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21808024,"kb_used_data":20313168,"kb_used_omap":961,"kb_used_meta":1493886,"kb_avail":72563816,"statfs":{"total":96636764160,"available":74305347584,"internally_reserved":0,"allocated":20800684032,"data_stored":23838092928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1529739301},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,8,14,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":2.8740000000000001,"5min":2.798,"15min":2.7360000000000002},"min":{"1min":0.22600000000000001,"5min":0.153,"15min":0.13800000000000001},"max":{"1min":15.569000000000001,"5min":28.088999999999999,"15min":28.088999999999999},"last":0.214},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.84799999999999998,"15min":1.369},"min":{"1min":0.189,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.4000000000000004,"5min":5.9660000000000002,"15min":14.116},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.1679999999999999,"15min":1.232},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":5.2850000000000001,"5min":11.577,"15min":24.614999999999998},"last":0.245},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.161,"15min":1.1619999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":1.417,"5min":6.7190000000000003,"15min":12.989000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359739170,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14154688,"kb_used_data":12859320,"kb_used_omap":597,"kb_used_meta":1294762,"kb_avail":80217152,"statfs":{"total":96636764160,"available":82142363648,"internally_reserved":0,"allocated":13167943680,"data_stored":14954003598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1325837217},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.212,"15min":1.3400000000000001},"min":{"1min":0.19800000000000001,"5min":0.16200000000000001,"15min":0.13900000000000001},"max":{"1min":6.601,"5min":9.8450000000000006,"15min":21.391999999999999},"last":4.9050000000000002},{"interface":"front","average":{"1min":1.379,"5min":1.2410000000000001,"15min":1.0229999999999999},"min":{"1min":0.23400000000000001,"5min":0.20200000000000001,"15min":0.182},"max":{"1min":11.045999999999999,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.38600000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0580000000000001,"15min":1.123},"min":{"1min":0.222,"5min":0.187,"15min":0.187},"max":{"1min":8.0410000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.97099999999999997,"15min":0.751},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":4.5819999999999999,"5min":6.827,"15min":8.077},"last":0.34999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12857470976,"data_stored":14643922856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20505448448,"data_stored":23542987920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7649284096,"data_stored":8900142312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309616640,"data_stored":309547308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297181184,"data_stored":297106998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606801920,"data_stored":606658432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:20:03.763 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-03T21:20:03.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:20:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:04.420 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:20:04.462 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2089,"stamp":"2023-10-03T21:20:04.089513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39803956805,"num_objects":14924,"num_object_clones":9472,"num_object_copies":29848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14796,"num_whiteouts":11,"num_read":1742511,"num_read_kb":6706157,"num_write":1698243,"num_write_kb":65928309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64709,"num_flush":14760,"num_flush_kb":59347401,"num_evict":9537,"num_evict_kb":38423097,"num_promote":9577,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120599,"ondisk_log_size":120599,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45000936,"kb_used_data":41288336,"kb_used_omap":1935,"kb_used_meta":3710640,"kb_avail":238114584,"statfs":{"total":289910292480,"available":243829334016,"internally_reserved":0,"allocated":42279256064,"data_stored":48344990019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":3799696227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,10,6,25,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":55865427,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5934,"num_read_kb":23516,"num_write":5944,"num_write_kb":264550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":83,"num_flush":56,"num_flush_kb":228840,"num_evict":28,"num_evict_kb":114344,"num_promote":33,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003388"},"pg_stats":[{"pgid":"2.7","version":"89'200830","reported_seq":402741,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.586182+0000","last_change":"2023-10-03T20:19:04.303179+0000","last_active":"2023-10-03T21:20:02.586182+0000","last_peered":"2023-10-03T21:20:02.586182+0000","last_clean":"2023-10-03T21:20:02.586182+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:20:02.586182+0000","last_undegraded":"2023-10-03T21:20:02.586182+0000","last_fullsized":"2023-10-03T21:20:02.586182+0000","mapping_epoch":12,"log_start":"86'190813","ondisk_log_start":"86'190813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:29:24.435845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4873199950,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":201749,"num_read_kb":787906,"num_write":199671,"num_write_kb":9436114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'200990","reported_seq":405172,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.613215+0000","last_change":"2023-10-03T20:19:04.232076+0000","last_active":"2023-10-03T21:20:02.613215+0000","last_peered":"2023-10-03T21:20:02.613215+0000","last_clean":"2023-10-03T21:20:02.613215+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:20:02.613215+0000","last_undegraded":"2023-10-03T21:20:02.613215+0000","last_fullsized":"2023-10-03T21:20:02.613215+0000","mapping_epoch":12,"log_start":"86'190913","ondisk_log_start":"86'190913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:27:33.334820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4934041919,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":204389,"num_read_kb":794624,"num_write":199814,"num_write_kb":8969312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'218340","reported_seq":438479,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.586642+0000","last_change":"2023-10-03T20:19:04.308490+0000","last_active":"2023-10-03T21:20:02.586642+0000","last_peered":"2023-10-03T21:20:02.586642+0000","last_clean":"2023-10-03T21:20:02.586642+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:20:02.586642+0000","last_undegraded":"2023-10-03T21:20:02.586642+0000","last_fullsized":"2023-10-03T21:20:02.586642+0000","mapping_epoch":12,"log_start":"86'208308","ondisk_log_start":"86'208308","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:03.359890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5153473015,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":220027,"num_read_kb":860441,"num_write":217094,"num_write_kb":8724581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'208060","reported_seq":419048,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.571178+0000","last_change":"2023-10-03T20:19:04.206725+0000","last_active":"2023-10-03T21:20:02.571178+0000","last_peered":"2023-10-03T21:20:02.571178+0000","last_clean":"2023-10-03T21:20:02.571178+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:20:02.571178+0000","last_undegraded":"2023-10-03T21:20:02.571178+0000","last_fullsized":"2023-10-03T21:20:02.571178+0000","mapping_epoch":12,"log_start":"86'198014","ondisk_log_start":"86'198014","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:08:51.601852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4915192291,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":210791,"num_read_kb":821698,"num_write":206844,"num_write_kb":7756889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'32045","reported_seq":52798,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:03.830482+0000","last_change":"2023-10-03T20:19:04.199607+0000","last_active":"2023-10-03T21:20:03.830482+0000","last_peered":"2023-10-03T21:20:03.830482+0000","last_clean":"2023-10-03T21:20:03.830482+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:20:03.830482+0000","last_undegraded":"2023-10-03T21:20:03.830482+0000","last_fullsized":"2023-10-03T21:20:03.830482+0000","mapping_epoch":15,"log_start":"84'22025","ondisk_log_start":"84'22025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:11:58.561538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":150564509,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18010,"num_read_kb":51751,"num_write":14787,"num_write_kb":49200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15963,"num_flush":4319,"num_flush_kb":17409416,"num_evict":2358,"num_evict_kb":9520877,"num_promote":2364,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'200563","reported_seq":402692,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:04.034172+0000","last_change":"2023-10-03T20:19:04.198050+0000","last_active":"2023-10-03T21:20:04.034172+0000","last_peered":"2023-10-03T21:20:04.034172+0000","last_clean":"2023-10-03T21:20:04.034172+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:20:04.034172+0000","last_undegraded":"2023-10-03T21:20:04.034172+0000","last_fullsized":"2023-10-03T21:20:04.034172+0000","mapping_epoch":12,"log_start":"86'190509","ondisk_log_start":"86'190509","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:46:58.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4667408705,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":201969,"num_read_kb":787677,"num_write":199420,"num_write_kb":6861724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'31909","reported_seq":52032,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.444952+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:20:02.444952+0000","last_peered":"2023-10-03T21:20:02.444952+0000","last_clean":"2023-10-03T21:20:02.444952+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:20:02.444952+0000","last_undegraded":"2023-10-03T21:20:02.444952+0000","last_fullsized":"2023-10-03T21:20:02.444952+0000","mapping_epoch":15,"log_start":"84'21830","ondisk_log_start":"84'21830","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:22:56.156359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150650864,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":18913,"num_read_kb":49542,"num_write":13796,"num_write_kb":44835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16167,"num_flush":3244,"num_flush_kb":13014580,"num_evict":2439,"num_evict_kb":9797696,"num_promote":2451,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"89'210225","reported_seq":422699,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:59.559319+0000","last_change":"2023-10-03T20:19:04.230032+0000","last_active":"2023-10-03T21:19:59.559319+0000","last_peered":"2023-10-03T21:19:59.559319+0000","last_clean":"2023-10-03T21:19:59.559319+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:19:59.559319+0000","last_undegraded":"2023-10-03T21:19:59.559319+0000","last_fullsized":"2023-10-03T21:19:59.559319+0000","mapping_epoch":12,"log_start":"86'200213","ondisk_log_start":"86'200213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:04:12.017189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5035999653,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":212315,"num_read_kb":829586,"num_write":209006,"num_write_kb":7576815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'28147","reported_seq":46869,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:03.240481+0000","last_change":"2023-10-03T20:19:04.198355+0000","last_active":"2023-10-03T21:20:03.240481+0000","last_peered":"2023-10-03T21:20:03.240481+0000","last_clean":"2023-10-03T21:20:03.240481+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:20:03.240481+0000","last_undegraded":"2023-10-03T21:20:03.240481+0000","last_fullsized":"2023-10-03T21:20:03.240481+0000","mapping_epoch":15,"log_start":"84'18130","ondisk_log_start":"84'18130","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:29:49.227097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":159036068,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":3,"num_read":16024,"num_read_kb":49576,"num_write":14139,"num_write_kb":46701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16809,"num_flush":3633,"num_flush_kb":14645356,"num_evict":2384,"num_evict_kb":9642453,"num_promote":2396,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"89'202971","reported_seq":408695,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:19:59.568412+0000","last_change":"2023-10-03T20:19:04.276516+0000","last_active":"2023-10-03T21:19:59.568412+0000","last_peered":"2023-10-03T21:19:59.568412+0000","last_clean":"2023-10-03T21:19:59.568412+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:19:59.568412+0000","last_undegraded":"2023-10-03T21:19:59.568412+0000","last_fullsized":"2023-10-03T21:19:59.568412+0000","mapping_epoch":12,"log_start":"86'192913","ondisk_log_start":"86'192913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:13:18.149220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4738806211,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":205525,"num_read_kb":802550,"num_write":201807,"num_write_kb":6857749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'39678","reported_seq":64425,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.234881+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:20:02.234881+0000","last_peered":"2023-10-03T21:20:02.234881+0000","last_clean":"2023-10-03T21:20:02.234881+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:20:02.234881+0000","last_undegraded":"2023-10-03T21:20:02.234881+0000","last_fullsized":"2023-10-03T21:20:02.234881+0000","mapping_epoch":15,"log_start":"84'29625","ondisk_log_start":"84'29625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:26:51.397031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154824292,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":24047,"num_read_kb":55241,"num_write":14072,"num_write_kb":46060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15770,"num_flush":3564,"num_flush_kb":14278049,"num_evict":2356,"num_evict_kb":9462071,"num_promote":2366,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'208844","reported_seq":417687,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:02.551002+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:20:02.551002+0000","last_peered":"2023-10-03T21:20:02.551002+0000","last_clean":"2023-10-03T21:20:02.551002+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:20:02.551002+0000","last_undegraded":"2023-10-03T21:20:02.551002+0000","last_fullsized":"2023-10-03T21:20:02.551002+0000","mapping_epoch":12,"log_start":"86'198807","ondisk_log_start":"86'198807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:28.300668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4870168960,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":208646,"num_read_kb":815474,"num_write":207656,"num_write_kb":9556435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":245,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T21:18:51.630010+0000","last_change":"2023-10-03T20:13:22.539837+0000","last_active":"2023-10-03T21:18:51.630010+0000","last_peered":"2023-10-03T21:18:51.630010+0000","last_clean":"2023-10-03T21:18:51.630010+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:18:51.630010+0000","last_undegraded":"2023-10-03T21:18:51.630010+0000","last_fullsized":"2023-10-03T21:18:51.630010+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:15:59.949043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":615075733,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":11,"num_read":76994,"num_read_kb":206110,"num_write":56794,"num_write_kb":186796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64709,"num_flush":14760,"num_flush_kb":59347401,"num_evict":9537,"num_evict_kb":38423097,"num_promote":9577,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1222012928,"data_stored":1221725973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40169,"ondisk_log_size":40169,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39188290704,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1665411,"num_read_kb":6499956,"num_write":1641312,"num_write_kb":65739619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41036890112,"data_stored":47103609120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80333,"ondisk_log_size":80333,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8954904,"kb_used_data":8063056,"kb_used_omap":376,"kb_used_meta":891463,"kb_avail":85416936,"statfs":{"total":96636764160,"available":87466942464,"internally_reserved":0,"allocated":8256569344,"data_stored":9506944514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":912859037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.98299999999999998,"15min":1.097},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.1589999999999998,"5min":7.907,"15min":19.114000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0349999999999999,"15min":0.98799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":6.1660000000000004,"5min":10.976000000000001,"15min":15.244},"last":0.42399999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.9749999999999996,"5min":30.677,"15min":30.677},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.0179999999999998,"5min":1.766,"15min":1.423},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.285,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21866568,"kb_used_data":20345344,"kb_used_omap":961,"kb_used_meta":1520254,"kb_avail":72505272,"statfs":{"total":96636764160,"available":74245398528,"internally_reserved":0,"allocated":20833632256,"data_stored":23866502814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1556740133},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1,17,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":2.8740000000000001,"5min":2.798,"15min":2.7360000000000002},"min":{"1min":0.22600000000000001,"5min":0.153,"15min":0.13800000000000001},"max":{"1min":15.569000000000001,"5min":28.088999999999999,"15min":28.088999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.84799999999999998,"15min":1.369},"min":{"1min":0.189,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.4000000000000004,"5min":5.9660000000000002,"15min":14.116},"last":0.32500000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.1679999999999999,"15min":1.232},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":5.2850000000000001,"5min":11.577,"15min":24.614999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.161,"15min":1.1619999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":1.417,"5min":6.7190000000000003,"15min":12.989000000000001},"last":0.43099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739171,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14179464,"kb_used_data":12879936,"kb_used_omap":597,"kb_used_meta":1298922,"kb_avail":80192376,"statfs":{"total":96636764160,"available":82116993024,"internally_reserved":0,"allocated":13189054464,"data_stored":14971542691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1330097057},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,5,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.212,"15min":1.3400000000000001},"min":{"1min":0.19800000000000001,"5min":0.16200000000000001,"15min":0.13900000000000001},"max":{"1min":6.601,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.2410000000000001,"15min":1.0229999999999999},"min":{"1min":0.23400000000000001,"5min":0.20200000000000001,"15min":0.182},"max":{"1min":11.045999999999999,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.24099999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0580000000000001,"15min":1.123},"min":{"1min":0.222,"5min":0.187,"15min":0.187},"max":{"1min":8.0410000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.97099999999999997,"15min":0.751},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":4.5819999999999999,"5min":6.827,"15min":8.077},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12869062656,"data_stored":14651598760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20518543360,"data_stored":23551868048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7649284096,"data_stored":8900142312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309665792,"data_stored":309596481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305545216,"data_stored":305471060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606801920,"data_stored":606658432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:20:04.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T21:20:05.292 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:05.293 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":89,"fsid":"0eaba888-2cf8-416c-91f9-3b34fc849c21","created":"2023-10-03T20:13:10.093612+0000","modified":"2023-10-03T21:19:06.615297+0000","last_up_change":"2023-10-03T20:13:20.015790+0000","last_in_change":"2023-10-03T20:13:11.030656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T20:13:20.822429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T20:13:25.447616+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":66,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cache","create_time":"2023-10-03T20:13:31.135339+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":66,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b76dced-3877-4565-b45a-3d1abd7b031c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6817","nonce":3168964859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6819","nonce":3168964859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6823","nonce":3168964859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3168964859},{"type":"v1","addr":"172.21.15.167:6821","nonce":3168964859}]},"public_addr":"172.21.15.167:6817/3168964859","cluster_addr":"172.21.15.167:6819/3168964859","heartbeat_back_addr":"172.21.15.167:6823/3168964859","heartbeat_front_addr":"172.21.15.167:6821/3168964859","state":["exists","up"]},{"osd":1,"uuid":"6404d2c7-ed12-4216-a92f-68b44d9d31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6801","nonce":1617383148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6803","nonce":1617383148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6813","nonce":1617383148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":1617383148},{"type":"v1","addr":"172.21.15.167:6808","nonce":1617383148}]},"public_addr":"172.21.15.167:6801/1617383148","cluster_addr":"172.21.15.167:6803/1617383148","heartbeat_back_addr":"172.21.15.167:6813/1617383148","heartbeat_front_addr":"172.21.15.167:6808/1617383148","state":["exists","up"]},{"osd":2,"uuid":"d8c3c345-38df-45e1-867c-559a9e29a2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6805","nonce":705294285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6809","nonce":705294285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6815","nonce":705294285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":705294285},{"type":"v1","addr":"172.21.15.167:6812","nonce":705294285}]},"public_addr":"172.21.15.167:6805/705294285","cluster_addr":"172.21.15.167:6809/705294285","heartbeat_back_addr":"172.21.15.167:6815/705294285","heartbeat_front_addr":"172.21.15.167:6812/705294285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.550984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.577924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T20:13:16.350491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2335679350":"2023-10-03T21:31:43.960393+0000","172.21.15.167:0/2447272347":"2023-10-03T21:31:29.406696+0000","172.21.15.167:0/584508052":"2023-10-03T21:29:06.507184+0000","172.21.15.167:0/593542325":"2023-10-03T21:21:26.949337+0000","172.21.15.167:0/3592042819":"2023-10-03T21:20:14.374679+0000","172.21.15.167:0/344627823":"2023-10-03T21:27:43.417936+0000","172.21.15.167:0/3020862503":"2023-10-03T21:23:52.025449+0000","172.21.15.167:0/1068382668":"2023-10-03T21:30:18.146610+0000","172.21.15.167:0/2415343388":"2023-10-03T21:22:39.522076+0000","172.21.15.167:0/2855518517":"2023-10-03T21:25:04.034332+0000","172.21.15.167:0/2386356822":"2023-10-03T21:26:30.614111+0000","172.21.15.167:0/4212689945":"2023-10-03T21:25:19.173978+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T21:20:06.344 INFO:tasks.ceph:Scrubbing osd.0 2023-10-03T21:20:06.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-03T21:20:06.666 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-03T21:20:06.666 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T21:20:06.666 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-03T21:20:06.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-03T21:20:07.608 INFO:teuthology.orchestra.run.smithi167.stderr:instructed osd(s) 0 to deep-scrub 2023-10-03T21:20:07.676 INFO:tasks.ceph:Scrubbing osd.1 2023-10-03T21:20:07.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-03T21:20:08.074 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-03T21:20:08.074 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T21:20:08.074 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-03T21:20:08.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-03T21:20:08.810 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:08.806+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:20:08.837 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:08.822+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:20:08.838 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:08.834+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:20:08.959 INFO:teuthology.orchestra.run.smithi167.stderr:instructed osd(s) 1 to deep-scrub 2023-10-03T21:20:09.029 INFO:tasks.ceph:Scrubbing osd.2 2023-10-03T21:20:09.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-03T21:20:09.435 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-03T21:20:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T21:20:09.436 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-03T21:20:09.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-03T21:20:09.868 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:09.862+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:20:09.882 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:09.878+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:20:09.882 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:09.878+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:20:09.882 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:09.878+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:20:09.882 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:09.878+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:20:09.882 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:09.878+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:20:10.258 INFO:teuthology.orchestra.run.smithi167.stderr:instructed osd(s) 2 to deep-scrub 2023-10-03T21:20:10.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:20:11.004 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:11.005 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:20:11.044 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2092,"stamp":"2023-10-03T21:20:10.090592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39794855493,"num_objects":14920,"num_object_clones":9472,"num_object_copies":29840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14789,"num_whiteouts":11,"num_read":1744958,"num_read_kb":6715833,"num_write":1700756,"num_write_kb":66040417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64709,"num_flush":14788,"num_flush_kb":59461877,"num_evict":9553,"num_evict_kb":38488461,"num_promote":9589,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120709,"ondisk_log_size":120709,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45006152,"kb_used_data":41258856,"kb_used_omap":1935,"kb_used_meta":3745328,"kb_avail":238109368,"statfs":{"total":289910292480,"available":243823992832,"internally_reserved":0,"allocated":42249068544,"data_stored":48311751000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":3835216739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,13,9,25,18,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19902515,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":4892,"num_read_kb":19360,"num_write":4977,"num_write_kb":207661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":51,"num_flush":57,"num_flush_kb":232984,"num_evict":32,"num_evict_kb":130732,"num_promote":31,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003397"},"pg_stats":[{"pgid":"2.7","version":"89'201058","reported_seq":403186,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:07.694190+0000","last_change":"2023-10-03T20:19:04.303179+0000","last_active":"2023-10-03T21:20:07.694190+0000","last_peered":"2023-10-03T21:20:07.694190+0000","last_clean":"2023-10-03T21:20:07.694190+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:20:07.694190+0000","last_undegraded":"2023-10-03T21:20:07.694190+0000","last_fullsized":"2023-10-03T21:20:07.694190+0000","mapping_epoch":12,"log_start":"86'191013","ondisk_log_start":"86'191013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:29:24.435845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4873580878,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":201966,"num_read_kb":788770,"num_write":199899,"num_write_kb":9449262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'201232","reported_seq":405649,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:07.650674+0000","last_change":"2023-10-03T20:19:04.232076+0000","last_active":"2023-10-03T21:20:07.650674+0000","last_peered":"2023-10-03T21:20:07.650674+0000","last_clean":"2023-10-03T21:20:07.650674+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:20:07.650674+0000","last_undegraded":"2023-10-03T21:20:07.650674+0000","last_fullsized":"2023-10-03T21:20:07.650674+0000","mapping_epoch":12,"log_start":"86'191213","ondisk_log_start":"86'191213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:27:33.334820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4934414655,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":204623,"num_read_kb":795544,"num_write":200056,"num_write_kb":8998888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'218586","reported_seq":438993,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:07.709433+0000","last_change":"2023-10-03T20:19:04.308490+0000","last_active":"2023-10-03T21:20:07.709433+0000","last_peered":"2023-10-03T21:20:07.709433+0000","last_clean":"2023-10-03T21:20:07.709433+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:20:07.709433+0000","last_undegraded":"2023-10-03T21:20:07.709433+0000","last_fullsized":"2023-10-03T21:20:07.709433+0000","mapping_epoch":12,"log_start":"86'208508","ondisk_log_start":"86'208508","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:03.359890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5157155319,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":220292,"num_read_kb":861493,"num_write":217340,"num_write_kb":8737813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'208312","reported_seq":419552,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:07.708107+0000","last_change":"2023-10-03T20:19:04.206725+0000","last_active":"2023-10-03T21:20:07.708107+0000","last_peered":"2023-10-03T21:20:07.708107+0000","last_clean":"2023-10-03T21:20:07.708107+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:20:07.708107+0000","last_undegraded":"2023-10-03T21:20:07.708107+0000","last_fullsized":"2023-10-03T21:20:07.708107+0000","mapping_epoch":12,"log_start":"86'198214","ondisk_log_start":"86'198214","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:08:51.601852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4915712483,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":211042,"num_read_kb":822678,"num_write":207096,"num_write_kb":7770149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'32067","reported_seq":52848,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:08.984226+0000","last_change":"2023-10-03T20:19:04.199607+0000","last_active":"2023-10-03T21:20:08.984226+0000","last_peered":"2023-10-03T21:20:08.984226+0000","last_clean":"2023-10-03T21:20:08.984226+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:20:08.984226+0000","last_undegraded":"2023-10-03T21:20:08.984226+0000","last_fullsized":"2023-10-03T21:20:08.984226+0000","mapping_epoch":15,"log_start":"84'22025","ondisk_log_start":"84'22025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158936733,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":18027,"num_read_kb":51819,"num_write":14797,"num_write_kb":49240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15963,"num_flush":4325,"num_flush_kb":17433948,"num_evict":2358,"num_evict_kb":9520877,"num_promote":2366,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'200797","reported_seq":403147,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:09.067801+0000","last_change":"2023-10-03T20:19:04.198050+0000","last_active":"2023-10-03T21:20:09.067801+0000","last_peered":"2023-10-03T21:20:09.067801+0000","last_clean":"2023-10-03T21:20:09.067801+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:20:09.067801+0000","last_undegraded":"2023-10-03T21:20:09.067801+0000","last_fullsized":"2023-10-03T21:20:09.067801+0000","mapping_epoch":12,"log_start":"86'190709","ondisk_log_start":"86'190709","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4667842881,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":202188,"num_read_kb":788545,"num_write":199654,"num_write_kb":6862660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'31953","reported_seq":52096,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:07.424081+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:20:07.424081+0000","last_peered":"2023-10-03T21:20:07.424081+0000","last_clean":"2023-10-03T21:20:07.424081+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:20:07.424081+0000","last_undegraded":"2023-10-03T21:20:07.424081+0000","last_fullsized":"2023-10-03T21:20:07.424081+0000","mapping_epoch":15,"log_start":"84'21930","ondisk_log_start":"84'21930","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:22:56.156359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":142233584,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":18924,"num_read_kb":49586,"num_write":13813,"num_write_kb":44883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16167,"num_flush":3248,"num_flush_kb":13030940,"num_evict":2444,"num_evict_kb":9818128,"num_promote":2454,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"89'210728","reported_seq":423690,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:09.671050+0000","last_change":"2023-10-03T20:19:04.230032+0000","last_active":"2023-10-03T21:20:09.671050+0000","last_peered":"2023-10-03T21:20:09.671050+0000","last_clean":"2023-10-03T21:20:09.671050+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:20:09.671050+0000","last_undegraded":"2023-10-03T21:20:09.671050+0000","last_fullsized":"2023-10-03T21:20:09.671050+0000","mapping_epoch":12,"log_start":"86'200713","ondisk_log_start":"86'200713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:04:12.017189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5036859813,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":212798,"num_read_kb":831490,"num_write":209509,"num_write_kb":7599239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'28178","reported_seq":46924,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:08.601772+0000","last_change":"2023-10-03T20:19:04.198355+0000","last_active":"2023-10-03T21:20:08.601772+0000","last_peered":"2023-10-03T21:20:08.601772+0000","last_clean":"2023-10-03T21:20:08.601772+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:20:08.601772+0000","last_undegraded":"2023-10-03T21:20:08.601772+0000","last_fullsized":"2023-10-03T21:20:08.601772+0000","mapping_epoch":15,"log_start":"84'18130","ondisk_log_start":"84'18130","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154854052,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16036,"num_read_kb":49624,"num_write":14158,"num_write_kb":46761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16809,"num_flush":3641,"num_flush_kb":14678048,"num_evict":2388,"num_evict_kb":9658801,"num_promote":2399,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"89'203455","reported_seq":409650,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:09.660491+0000","last_change":"2023-10-03T20:19:04.276516+0000","last_active":"2023-10-03T21:20:09.660491+0000","last_peered":"2023-10-03T21:20:09.660491+0000","last_clean":"2023-10-03T21:20:09.660491+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:20:09.660491+0000","last_undegraded":"2023-10-03T21:20:09.660491+0000","last_fullsized":"2023-10-03T21:20:09.660491+0000","mapping_epoch":12,"log_start":"86'193413","ondisk_log_start":"86'193413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:13:18.149220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4739748291,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":205994,"num_read_kb":804414,"num_write":202291,"num_write_kb":6863777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'39749","reported_seq":64520,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:06.949448+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:20:06.949448+0000","last_peered":"2023-10-03T21:20:06.949448+0000","last_clean":"2023-10-03T21:20:06.949448+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:20:06.949448+0000","last_undegraded":"2023-10-03T21:20:06.949448+0000","last_fullsized":"2023-10-03T21:20:06.949448+0000","mapping_epoch":15,"log_start":"84'29725","ondisk_log_start":"84'29725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:26:51.397031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":142306916,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":24053,"num_read_kb":55265,"num_write":14097,"num_write_kb":46132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15770,"num_flush":3574,"num_flush_kb":14318941,"num_evict":2363,"num_evict_kb":9490655,"num_promote":2370,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'209097","reported_seq":418205,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:07.701301+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:20:07.701301+0000","last_peered":"2023-10-03T21:20:07.701301+0000","last_clean":"2023-10-03T21:20:07.701301+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:20:07.701301+0000","last_undegraded":"2023-10-03T21:20:07.701301+0000","last_fullsized":"2023-10-03T21:20:07.701301+0000","mapping_epoch":12,"log_start":"86'199007","ondisk_log_start":"86'199007","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:28.300668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4870619520,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":208909,"num_read_kb":816514,"num_write":207909,"num_write_kb":9569719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":245,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T21:18:51.630010+0000","last_change":"2023-10-03T20:13:22.539837+0000","last_active":"2023-10-03T21:18:51.630010+0000","last_peered":"2023-10-03T21:18:51.630010+0000","last_clean":"2023-10-03T21:18:51.630010+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:18:51.630010+0000","last_undegraded":"2023-10-03T21:18:51.630010+0000","last_fullsized":"2023-10-03T21:18:51.630010+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:15:59.949043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":598331285,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":11,"num_read":77040,"num_read_kb":206294,"num_write":56865,"num_write_kb":187016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64709,"num_flush":14788,"num_flush_kb":59461877,"num_evict":9553,"num_evict_kb":38488461,"num_promote":9589,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1196843008,"data_stored":1196556074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40137,"ondisk_log_size":40137,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39195933840,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1667812,"num_read_kb":6509448,"num_write":1643754,"num_write_kb":65851507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41058111488,"data_stored":47122057504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80475,"ondisk_log_size":80475,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739170,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8964872,"kb_used_data":8064704,"kb_used_omap":376,"kb_used_meta":899783,"kb_avail":85406968,"statfs":{"total":96636764160,"available":87456735232,"internally_reserved":0,"allocated":8258256896,"data_stored":9507743255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":921378717},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.98299999999999998,"15min":1.097},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.1589999999999998,"5min":7.907,"15min":19.114000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0349999999999999,"15min":0.98799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":6.1660000000000004,"5min":10.976000000000001,"15min":15.244},"last":0.31900000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.9749999999999996,"5min":30.677,"15min":30.677},"last":0.436},{"interface":"front","average":{"1min":3.0179999999999998,"5min":1.766,"15min":1.423},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.285,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739161,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21869728,"kb_used_data":20326288,"kb_used_omap":961,"kb_used_meta":1542462,"kb_avail":72502112,"statfs":{"total":96636764160,"available":74242162688,"internally_reserved":0,"allocated":20814118912,"data_stored":23845809822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1579481125},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,9,6,12,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":2.8740000000000001,"5min":2.798,"15min":2.7360000000000002},"min":{"1min":0.22600000000000001,"5min":0.153,"15min":0.13800000000000001},"max":{"1min":15.569000000000001,"5min":28.088999999999999,"15min":28.088999999999999},"last":0.247},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.84799999999999998,"15min":1.369},"min":{"1min":0.189,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.4000000000000004,"5min":5.9660000000000002,"15min":14.116},"last":0.19900000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:19:19 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.1679999999999999,"15min":1.232},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":5.2850000000000001,"5min":11.577,"15min":24.614999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.161,"15min":1.1619999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":1.417,"5min":6.7190000000000003,"15min":12.989000000000001},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739172,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14171552,"kb_used_data":12867864,"kb_used_omap":597,"kb_used_meta":1303082,"kb_avail":80200288,"statfs":{"total":96636764160,"available":82125094912,"internally_reserved":0,"allocated":13176692736,"data_stored":14958197923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1334356897},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,9,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.212,"15min":1.3400000000000001},"min":{"1min":0.19800000000000001,"5min":0.16200000000000001,"15min":0.13900000000000001},"max":{"1min":6.601,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.2410000000000001,"15min":1.0229999999999999},"min":{"1min":0.23400000000000001,"5min":0.20200000000000001,"15min":0.182},"max":{"1min":11.045999999999999,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:19:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0580000000000001,"15min":1.123},"min":{"1min":0.222,"5min":0.187,"15min":0.187},"max":{"1min":8.0410000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":4.9240000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.97099999999999997,"15min":0.751},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":4.5819999999999999,"5min":6.827,"15min":8.077},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12874481664,"data_stored":14656370600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20528476160,"data_stored":23560555664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7655153664,"data_stored":8905131240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":318001152,"data_stored":317931841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284610560,"data_stored":284536404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":594231296,"data_stored":594087829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:20:11.047 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.047 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.048 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.048 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.048 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.048 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.048 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.049 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.049 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.049 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.049 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.050 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.050 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:11.050 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:20:20.740 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:20.737+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:20:27.612 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.605+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:20:27.626 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.621+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:20:27.639 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.637+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:20:27.653 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.649+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:20:27.714 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:20:27.714 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:20:27.715 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:20:27.715 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:20:27.715 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:20:27.715 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:20:27.715 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:27.709+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:20:31.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:20:32.037 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:32.037 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:20:32.076 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2103,"stamp":"2023-10-03T21:20:30.099843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39849570032,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":1752418,"num_read_kb":6745353,"num_write":1708216,"num_write_kb":66347718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64880,"num_flush":14856,"num_flush_kb":59739817,"num_evict":9603,"num_evict_kb":38692653,"num_promote":9640,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120828,"ondisk_log_size":120828,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45177912,"kb_used_data":41370584,"kb_used_omap":1935,"kb_used_meta":3801264,"kb_avail":237937608,"statfs":{"total":289910292480,"available":243648110592,"internally_reserved":0,"allocated":42363478016,"data_stored":48404668925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":3892495203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,2,4,5,11,13,11,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30466107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":4347,"num_read_kb":17180,"num_write":4376,"num_write_kb":172565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":59,"num_flush":43,"num_flush_kb":175720,"num_evict":28,"num_evict_kb":114404,"num_promote":29,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007946"},"pg_stats":[{"pgid":"2.7","version":"90'201980","reported_seq":405055,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:28.042212+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:20:28.042212+0000","last_peered":"2023-10-03T21:20:28.042212+0000","last_clean":"2023-10-03T21:20:28.042212+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:20:28.042212+0000","last_undegraded":"2023-10-03T21:20:28.042212+0000","last_fullsized":"2023-10-03T21:20:28.042212+0000","mapping_epoch":12,"log_start":"86'191913","ondisk_log_start":"86'191913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4881547598,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":202908,"num_read_kb":792494,"num_write":200821,"num_write_kb":9481498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'202073","reported_seq":407368,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:28.011842+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:20:28.011842+0000","last_peered":"2023-10-03T21:20:28.011842+0000","last_clean":"2023-10-03T21:20:28.011842+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:20:28.011842+0000","last_undegraded":"2023-10-03T21:20:28.011842+0000","last_fullsized":"2023-10-03T21:20:28.011842+0000","mapping_epoch":12,"log_start":"86'192013","ondisk_log_start":"86'192013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4942397759,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":205492,"num_read_kb":798988,"num_write":200897,"num_write_kb":9043104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'219557","reported_seq":440919,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:28.027875+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:20:28.027875+0000","last_peered":"2023-10-03T21:20:28.027875+0000","last_clean":"2023-10-03T21:20:28.027875+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:20:28.027875+0000","last_undegraded":"2023-10-03T21:20:28.027875+0000","last_fullsized":"2023-10-03T21:20:28.027875+0000","mapping_epoch":12,"log_start":"86'209508","ondisk_log_start":"86'209508","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5165109751,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":221239,"num_read_kb":865249,"num_write":218311,"num_write_kb":8790701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'209248","reported_seq":421436,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:28.035595+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:20:28.035595+0000","last_peered":"2023-10-03T21:20:28.035595+0000","last_clean":"2023-10-03T21:20:28.035595+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:20:28.035595+0000","last_undegraded":"2023-10-03T21:20:28.035595+0000","last_fullsized":"2023-10-03T21:20:28.035595+0000","mapping_epoch":12,"log_start":"86'199214","ondisk_log_start":"86'199214","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":16,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4923347427,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":211972,"num_read_kb":826354,"num_write":208021,"num_write_kb":7822893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'32232","reported_seq":53121,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:28.839314+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:20:28.839314+0000","last_peered":"2023-10-03T21:20:28.839314+0000","last_clean":"2023-10-03T21:20:28.839314+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:20:28.839314+0000","last_undegraded":"2023-10-03T21:20:28.839314+0000","last_fullsized":"2023-10-03T21:20:28.839314+0000","mapping_epoch":15,"log_start":"84'22225","ondisk_log_start":"84'22225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154853087,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18081,"num_read_kb":52035,"num_write":14869,"num_write_kb":49474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16029,"num_flush":4342,"num_flush_kb":17503356,"num_evict":2371,"num_evict_kb":9573913,"num_promote":2378,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'201799","reported_seq":405078,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:29.302960+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:20:29.302960+0000","last_peered":"2023-10-03T21:20:29.302960+0000","last_clean":"2023-10-03T21:20:29.302960+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:20:29.302960+0000","last_undegraded":"2023-10-03T21:20:29.302960+0000","last_fullsized":"2023-10-03T21:20:29.302960+0000","mapping_epoch":12,"log_start":"86'191709","ondisk_log_start":"86'191709","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":173,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 20s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4672581953,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":203094,"num_read_kb":792121,"num_write":200540,"num_write_kb":6886620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'32094","reported_seq":52322,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:27.617370+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:20:27.617370+0000","last_peered":"2023-10-03T21:20:27.617370+0000","last_clean":"2023-10-03T21:20:27.617370+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:20:27.617370+0000","last_undegraded":"2023-10-03T21:20:27.617370+0000","last_fullsized":"2023-10-03T21:20:27.617370+0000","mapping_epoch":15,"log_start":"84'22030","ondisk_log_start":"84'22030","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150585374,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":18972,"num_read_kb":49778,"num_write":13871,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16213,"num_flush":3264,"num_flush_kb":13096384,"num_evict":2458,"num_evict_kb":9875368,"num_promote":2470,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"90'211682","reported_seq":425616,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:29.833683+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:20:29.833683+0000","last_peered":"2023-10-03T21:20:29.833683+0000","last_clean":"2023-10-03T21:20:29.833683+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:20:29.833683+0000","last_undegraded":"2023-10-03T21:20:29.833683+0000","last_fullsized":"2023-10-03T21:20:29.833683+0000","mapping_epoch":12,"log_start":"86'201614","ondisk_log_start":"86'201614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":109,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 17s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5041394085,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":213748,"num_read_kb":835246,"num_write":210387,"num_write_kb":7635427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'28325","reported_seq":47162,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:29.302603+0000","last_change":"2023-10-03T21:20:13.072884+0000","last_active":"2023-10-03T21:20:29.302603+0000","last_peered":"2023-10-03T21:20:29.302603+0000","last_clean":"2023-10-03T21:20:29.302603+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:20:29.302603+0000","last_undegraded":"2023-10-03T21:20:29.302603+0000","last_fullsized":"2023-10-03T21:20:29.302603+0000","mapping_epoch":15,"log_start":"84'18230","ondisk_log_start":"84'18230","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":26,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 16s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146440908,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16082,"num_read_kb":49808,"num_write":14230,"num_write_kb":46988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16849,"num_flush":3661,"num_flush_kb":14759808,"num_evict":2403,"num_evict_kb":9720089,"num_promote":2412,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"90'204379","reported_seq":411450,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:29.840456+0000","last_change":"2023-10-03T21:20:27.875276+0000","last_active":"2023-10-03T21:20:29.840456+0000","last_peered":"2023-10-03T21:20:29.840456+0000","last_clean":"2023-10-03T21:20:29.840456+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:20:29.840456+0000","last_undegraded":"2023-10-03T21:20:29.840456+0000","last_fullsized":"2023-10-03T21:20:29.840456+0000","mapping_epoch":12,"log_start":"86'194313","ondisk_log_start":"86'194313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4747964867,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":206855,"num_read_kb":807810,"num_write":203215,"num_write_kb":6883809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'39913","reported_seq":64779,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:27.719967+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:20:27.719967+0000","last_peered":"2023-10-03T21:20:27.719967+0000","last_clean":"2023-10-03T21:20:27.719967+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:20:27.719967+0000","last_undegraded":"2023-10-03T21:20:27.719967+0000","last_fullsized":"2023-10-03T21:20:27.719967+0000","mapping_epoch":15,"log_start":"84'29825","ondisk_log_start":"84'29825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150613623,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":24100,"num_read_kb":55453,"num_write":14155,"num_write_kb":46330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15789,"num_flush":3589,"num_flush_kb":14380269,"num_evict":2371,"num_evict_kb":9523283,"num_promote":2380,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'209950","reported_seq":419926,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:28.030586+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:20:28.030586+0000","last_peered":"2023-10-03T21:20:28.030586+0000","last_clean":"2023-10-03T21:20:28.030586+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:20:28.030586+0000","last_undegraded":"2023-10-03T21:20:28.030586+0000","last_fullsized":"2023-10-03T21:20:28.030586+0000","mapping_epoch":12,"log_start":"86'199907","ondisk_log_start":"86'199907","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4872143232,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":209769,"num_read_kb":819926,"num_write":208762,"num_write_kb":9613923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":248,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-03T21:20:14.145276+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:20:14.145276+0000","last_peered":"2023-10-03T21:20:14.145276+0000","last_clean":"2023-10-03T21:20:14.145276+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:20:14.145276+0000","last_undegraded":"2023-10-03T21:20:14.145276+0000","last_fullsized":"2023-10-03T21:20:14.145276+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602492992,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":77235,"num_read_kb":207074,"num_write":57125,"num_write_kb":187849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64880,"num_flush":14856,"num_flush_kb":59739817,"num_evict":9603,"num_evict_kb":38692653,"num_promote":9640,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1209450496,"data_stored":1209163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40254,"ondisk_log_size":40254,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39246486672,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1675077,"num_read_kb":6538188,"num_write":1650954,"num_write_kb":66157975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41159798784,"data_stored":47202302240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80477,"ondisk_log_size":80477,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739174,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8992672,"kb_used_data":8078680,"kb_used_omap":376,"kb_used_meta":913607,"kb_avail":85379168,"statfs":{"total":96636764160,"available":87428268032,"internally_reserved":0,"allocated":8272568320,"data_stored":9518405255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":935534493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.98299999999999998,"15min":1.097},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.1589999999999998,"5min":7.907,"15min":19.114000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0349999999999999,"15min":0.98799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":6.1660000000000004,"5min":10.976000000000001,"15min":15.244},"last":0.60099999999999998}]},{"osd":1,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.9749999999999996,"5min":30.677,"15min":30.677},"last":0.629},{"interface":"front","average":{"1min":3.0179999999999998,"5min":1.766,"15min":1.423},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.285,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.55500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739165,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21972160,"kb_used_data":20392176,"kb_used_omap":961,"kb_used_meta":1579006,"kb_avail":72399680,"statfs":{"total":96636764160,"available":74137272320,"internally_reserved":0,"allocated":20881588224,"data_stored":23902535332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1616902181},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,2,3,3,5,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2,"15min":2.6960000000000002},"min":{"1min":0.183,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":5.9269999999999996,"5min":17.309999999999999,"15min":28.088999999999999},"last":0.879},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.014,"15min":1.4139999999999999},"min":{"1min":0.19900000000000001,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.8140000000000001,"5min":5.9660000000000002,"15min":14.116},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.2889999999999999,"15min":1.278},"min":{"1min":0.20499999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":7.9539999999999997,"5min":11.577,"15min":24.614999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.4159999999999999,"15min":1.2509999999999999},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":14.201000000000001,"5min":14.201000000000001,"15min":14.201000000000001},"last":4.6909999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739176,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14213080,"kb_used_data":12899728,"kb_used_omap":597,"kb_used_meta":1308650,"kb_avail":80158760,"statfs":{"total":96636764160,"available":82082570240,"internally_reserved":0,"allocated":13209321472,"data_stored":14983728338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1340058529},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,0,6,3,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.153,"15min":1.331},"min":{"1min":0.19,"5min":0.17499999999999999,"15min":0.13900000000000001},"max":{"1min":4.9050000000000002,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.276,"15min":1.0129999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":5.2670000000000003,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.41699999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95799999999999996,"15min":1.145},"min":{"1min":0.22600000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.4880000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.872,"5min":6.5709999999999997,"15min":6.827},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12910940160,"data_stored":14685779880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20579426304,"data_stored":23600774288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7669432320,"data_stored":8915748072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301359104,"data_stored":301289899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301268992,"data_stored":301194901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606822400,"data_stored":606679104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:20:32.078 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.078 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.078 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.078 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.079 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.079 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.079 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.079 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.079 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.080 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.080 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.080 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:32.080 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:20:45.137 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.133+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:20:45.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:20:45.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:20:45.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:20:45.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.137+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:20:45.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.145+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:20:45.150 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:45.145+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:20:49.540 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:49.529+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:20:50.005 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:50.001+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:20:50.048 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:50.033+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:20:50.048 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:20:50.033+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:20:50.479 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:20:50.473+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:20:52.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:20:52.886 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:20:52.887 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:20:52.929 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2114,"stamp":"2023-10-03T21:20:52.107686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39905111685,"num_objects":14923,"num_object_clones":9472,"num_object_copies":29846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14790,"num_whiteouts":11,"num_read":1759831,"num_read_kb":6774705,"num_write":1715751,"num_write_kb":66638762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64773,"num_flush":14918,"num_flush_kb":59992997,"num_evict":9651,"num_evict_kb":38888729,"num_promote":9690,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120767,"ondisk_log_size":120767,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45435272,"kb_used_data":41480368,"kb_used_omap":1935,"kb_used_meta":3952944,"kb_avail":237680248,"statfs":{"total":289910292480,"available":243384573952,"internally_reserved":0,"allocated":42475896832,"data_stored":48493756711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4047815523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,16,15,29,22,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10239927,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":3702,"num_read_kb":14648,"num_write":3789,"num_write_kb":149658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-73,"num_flush":30,"num_flush_kb":122484,"num_evict":28,"num_evict_kb":114416,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":-1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002058"},"pg_stats":[{"pgid":"2.7","version":"90'202850","reported_seq":406766,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:48.438316+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:20:48.438316+0000","last_peered":"2023-10-03T21:20:48.438316+0000","last_clean":"2023-10-03T21:20:48.438316+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:20:48.438316+0000","last_undegraded":"2023-10-03T21:20:48.438316+0000","last_fullsized":"2023-10-03T21:20:48.438316+0000","mapping_epoch":12,"log_start":"86'192813","ondisk_log_start":"86'192813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4886081870,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":203739,"num_read_kb":795794,"num_write":201691,"num_write_kb":9517658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'202962","reported_seq":409169,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:48.405017+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:20:48.405017+0000","last_peered":"2023-10-03T21:20:48.405017+0000","last_clean":"2023-10-03T21:20:48.405017+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:20:48.405017+0000","last_undegraded":"2023-10-03T21:20:48.405017+0000","last_fullsized":"2023-10-03T21:20:48.405017+0000","mapping_epoch":12,"log_start":"86'192913","ondisk_log_start":"86'192913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4952777023,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":206397,"num_read_kb":802560,"num_write":201786,"num_write_kb":9099744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'220485","reported_seq":442812,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:48.405238+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:20:48.405238+0000","last_peered":"2023-10-03T21:20:48.405238+0000","last_clean":"2023-10-03T21:20:48.405238+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:20:48.405238+0000","last_undegraded":"2023-10-03T21:20:48.405238+0000","last_fullsized":"2023-10-03T21:20:48.405238+0000","mapping_epoch":12,"log_start":"86'210408","ondisk_log_start":"86'210408","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5166690807,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":222198,"num_read_kb":869053,"num_write":219239,"num_write_kb":8814825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'210286","reported_seq":423404,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:48.413280+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:20:48.413280+0000","last_peered":"2023-10-03T21:20:48.413280+0000","last_clean":"2023-10-03T21:20:48.413280+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:20:48.413280+0000","last_undegraded":"2023-10-03T21:20:48.413280+0000","last_fullsized":"2023-10-03T21:20:48.413280+0000","mapping_epoch":12,"log_start":"86'200214","ondisk_log_start":"86'200214","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":223,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 25s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4931379683,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":212885,"num_read_kb":829970,"num_write":208919,"num_write_kb":7850917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'32379","reported_seq":53361,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:49.060444+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:20:49.060444+0000","last_peered":"2023-10-03T21:20:49.060444+0000","last_clean":"2023-10-03T21:20:49.060444+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:20:49.060444+0000","last_undegraded":"2023-10-03T21:20:49.060444+0000","last_fullsized":"2023-10-03T21:20:49.060444+0000","mapping_epoch":15,"log_start":"84'22325","ondisk_log_start":"84'22325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159030963,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":18141,"num_read_kb":52275,"num_write":14934,"num_write_kb":49688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15985,"num_flush":4360,"num_flush_kb":17576852,"num_evict":2382,"num_evict_kb":9618881,"num_promote":2390,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'202834","reported_seq":407026,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:49.416561+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:20:49.416561+0000","last_peered":"2023-10-03T21:20:49.416561+0000","last_clean":"2023-10-03T21:20:49.416561+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:20:49.416561+0000","last_undegraded":"2023-10-03T21:20:49.416561+0000","last_fullsized":"2023-10-03T21:20:49.416561+0000","mapping_epoch":12,"log_start":"86'192809","ondisk_log_start":"86'192809","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":316,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 40s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4674408769,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":203989,"num_read_kb":795665,"num_write":201480,"num_write_kb":6898540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'32217","reported_seq":52530,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:47.883896+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:20:47.883896+0000","last_peered":"2023-10-03T21:20:47.883896+0000","last_clean":"2023-10-03T21:20:47.883896+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:20:47.883896+0000","last_undegraded":"2023-10-03T21:20:47.883896+0000","last_fullsized":"2023-10-03T21:20:47.883896+0000","mapping_epoch":15,"log_start":"84'22130","ondisk_log_start":"84'22130","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154783779,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19024,"num_read_kb":49986,"num_write":13936,"num_write_kb":45275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16218,"num_flush":3276,"num_flush_kb":13145352,"num_evict":2468,"num_evict_kb":9916204,"num_promote":2481,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"90'212655","reported_seq":427524,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:20:49.996636+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:20:49.996636+0000","last_peered":"2023-10-03T21:20:49.996636+0000","last_clean":"2023-10-03T21:20:49.996636+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:20:49.996636+0000","last_undegraded":"2023-10-03T21:20:49.996636+0000","last_fullsized":"2023-10-03T21:20:49.996636+0000","mapping_epoch":12,"log_start":"86'202614","ondisk_log_start":"86'202614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":198,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 37s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5055537573,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":214665,"num_read_kb":838866,"num_write":211297,"num_write_kb":7687999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'28459","reported_seq":47394,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:49.084282+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:20:49.084282+0000","last_peered":"2023-10-03T21:20:49.084282+0000","last_clean":"2023-10-03T21:20:49.084282+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:20:49.084282+0000","last_undegraded":"2023-10-03T21:20:49.084282+0000","last_fullsized":"2023-10-03T21:20:49.084282+0000","mapping_epoch":15,"log_start":"84'18430","ondisk_log_start":"84'18430","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150680195,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":3,"num_read":16137,"num_read_kb":50028,"num_write":14283,"num_write_kb":47146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16776,"num_flush":3678,"num_flush_kb":14829220,"num_evict":2416,"num_evict_kb":9773185,"num_promote":2426,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"90'205285","reported_seq":413237,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:50.038784+0000","last_change":"2023-10-03T21:20:48.020435+0000","last_active":"2023-10-03T21:20:50.038784+0000","last_peered":"2023-10-03T21:20:50.038784+0000","last_clean":"2023-10-03T21:20:50.038784+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:20:50.038784+0000","last_undegraded":"2023-10-03T21:20:50.038784+0000","last_fullsized":"2023-10-03T21:20:50.038784+0000","mapping_epoch":12,"log_start":"86'195213","ondisk_log_start":"86'195213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4752716227,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":207722,"num_read_kb":811246,"num_write":204121,"num_write_kb":6920093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'40069","reported_seq":65035,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:47.904403+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:20:47.904403+0000","last_peered":"2023-10-03T21:20:47.904403+0000","last_clean":"2023-10-03T21:20:47.904403+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:20:47.904403+0000","last_undegraded":"2023-10-03T21:20:47.904403+0000","last_fullsized":"2023-10-03T21:20:47.904403+0000","mapping_epoch":15,"log_start":"84'30025","ondisk_log_start":"84'30025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":146407036,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":24154,"num_read_kb":55669,"num_write":14226,"num_write_kb":46556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15794,"num_flush":3604,"num_flush_kb":14441573,"num_evict":2385,"num_evict_kb":9580459,"num_promote":2393,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'210890","reported_seq":421778,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:48.401304+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:20:48.401304+0000","last_peered":"2023-10-03T21:20:48.401304+0000","last_clean":"2023-10-03T21:20:48.401304+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:20:48.401304+0000","last_undegraded":"2023-10-03T21:20:48.401304+0000","last_fullsized":"2023-10-03T21:20:48.401304+0000","mapping_epoch":12,"log_start":"86'200807","ondisk_log_start":"86'200807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4874027392,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":210674,"num_read_kb":823502,"num_write":209702,"num_write_kb":9658427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":249,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:16.661774+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:20:16.661774+0000","last_peered":"2023-10-03T21:20:16.661774+0000","last_clean":"2023-10-03T21:20:16.661774+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:20:16.661774+0000","last_undegraded":"2023-10-03T21:20:16.661774+0000","last_fullsized":"2023-10-03T21:20:16.661774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610901973,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":11,"num_read":77456,"num_read_kb":207958,"num_write":57379,"num_write_kb":188665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64773,"num_flush":14918,"num_flush_kb":59992997,"num_evict":9651,"num_evict_kb":38888729,"num_promote":9690,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1222098944,"data_stored":1221812138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40214,"ondisk_log_size":40214,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39293619344,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1682269,"num_read_kb":6566656,"num_write":1658235,"num_write_kb":66448203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41254957056,"data_stored":47274232096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80456,"ondisk_log_size":80456,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739178,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9057708,"kb_used_data":8107620,"kb_used_omap":376,"kb_used_meta":949703,"kb_avail":85314132,"statfs":{"total":96636764160,"available":87361671168,"internally_reserved":0,"allocated":8302202880,"data_stored":9542538780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":972496797},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.98299999999999998,"15min":1.097},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.1589999999999998,"5min":7.907,"15min":19.114000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0349999999999999,"15min":0.98799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":6.1660000000000004,"5min":10.976000000000001,"15min":15.244},"last":0.26400000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.9749999999999996,"5min":30.677,"15min":30.677},"last":0.57599999999999996},{"interface":"front","average":{"1min":3.0179999999999998,"5min":1.766,"15min":1.423},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.285,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.36699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739169,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22098460,"kb_used_data":20438932,"kb_used_omap":961,"kb_used_meta":1658558,"kb_avail":72273380,"statfs":{"total":96636764160,"available":74007941120,"internally_reserved":0,"allocated":20929466368,"data_stored":23938743982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1698363429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,10,8,12,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2,"15min":2.6960000000000002},"min":{"1min":0.183,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":5.9269999999999996,"5min":17.309999999999999,"15min":28.088999999999999},"last":3.2959999999999998},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.014,"15min":1.4139999999999999},"min":{"1min":0.19900000000000001,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.8140000000000001,"5min":5.9660000000000002,"15min":14.116},"last":0.36499999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.2889999999999999,"15min":1.278},"min":{"1min":0.20499999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":7.9539999999999997,"5min":11.577,"15min":24.614999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.4159999999999999,"15min":1.2509999999999999},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":14.201000000000001,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739180,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14279104,"kb_used_data":12933816,"kb_used_omap":597,"kb_used_meta":1344682,"kb_avail":80092736,"statfs":{"total":96636764160,"available":82014961664,"internally_reserved":0,"allocated":13244227584,"data_stored":15012473949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1376955297},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,6,7,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.153,"15min":1.331},"min":{"1min":0.19,"5min":0.17499999999999999,"15min":0.13900000000000001},"max":{"1min":4.9050000000000002,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.441},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.276,"15min":1.0129999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":5.2670000000000003,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95799999999999996,"15min":1.145},"min":{"1min":0.22600000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.4880000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":4.0780000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.872,"5min":6.5709999999999997,"15min":6.827},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12937453568,"data_stored":14706137000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20627103744,"data_stored":23636823184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7690399744,"data_stored":8931271912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309776384,"data_stored":309707062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301260800,"data_stored":301186719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":611061760,"data_stored":610918357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:20:52.931 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.931 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.931 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.931 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.931 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.932 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.932 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.932 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.932 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.932 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.932 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:20:52.933 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:21:03.883 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:03.877+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:21:03.910 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:03.905+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:21:12.826 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:12.820+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:21:12.867 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:12.860+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:21:12.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:21:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:21:13.926 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:21:13.968 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2124,"stamp":"2023-10-03T21:21:12.117247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39926607474,"num_objects":14919,"num_object_clones":9472,"num_object_copies":29838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1767509,"num_read_kb":6805141,"num_write":1723296,"num_write_kb":66917751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64754,"num_flush":14981,"num_flush_kb":60250437,"num_evict":9701,"num_evict_kb":39093021,"num_promote":9736,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120609,"ondisk_log_size":120609,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45566912,"kb_used_data":41530088,"kb_used_omap":1935,"kb_used_meta":4034864,"kb_avail":237548608,"statfs":{"total":289910292480,"available":243249774592,"internally_reserved":0,"allocated":42526810112,"data_stored":48529395969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4131701603},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,6,14,24,17,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20922376,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4312,"num_read_kb":17108,"num_write":4273,"num_write_kb":139439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":8,"num_flush":24,"num_flush_kb":98032,"num_evict":23,"num_evict_kb":93980,"num_promote":24,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005842"},"pg_stats":[{"pgid":"2.7","version":"90'203738","reported_seq":408548,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:08.814488+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:21:08.814488+0000","last_peered":"2023-10-03T21:21:08.814488+0000","last_clean":"2023-10-03T21:21:08.814488+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:21:08.814488+0000","last_undegraded":"2023-10-03T21:21:08.814488+0000","last_fullsized":"2023-10-03T21:21:08.814488+0000","mapping_epoch":12,"log_start":"86'193713","ondisk_log_start":"86'193713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4890755406,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":204626,"num_read_kb":799306,"num_write":202579,"num_write_kb":9562054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'203870","reported_seq":410983,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:08.858328+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:21:08.858328+0000","last_peered":"2023-10-03T21:21:08.858328+0000","last_clean":"2023-10-03T21:21:08.858328+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:21:08.858328+0000","last_undegraded":"2023-10-03T21:21:08.858328+0000","last_fullsized":"2023-10-03T21:21:08.858328+0000","mapping_epoch":12,"log_start":"86'193813","ondisk_log_start":"86'193813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4963803455,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":207298,"num_read_kb":806108,"num_write":202694,"num_write_kb":9136040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'221470","reported_seq":444821,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:08.845959+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:21:08.845959+0000","last_peered":"2023-10-03T21:21:08.845959+0000","last_clean":"2023-10-03T21:21:08.845959+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:21:08.845959+0000","last_undegraded":"2023-10-03T21:21:08.845959+0000","last_fullsized":"2023-10-03T21:21:08.845959+0000","mapping_epoch":12,"log_start":"87'211408","ondisk_log_start":"87'211408","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5171626487,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":223211,"num_read_kb":873061,"num_write":220224,"num_write_kb":8826945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'211334","reported_seq":425443,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:08.825283+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:21:08.825283+0000","last_peered":"2023-10-03T21:21:08.825283+0000","last_clean":"2023-10-03T21:21:08.825283+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:21:08.825283+0000","last_undegraded":"2023-10-03T21:21:08.825283+0000","last_fullsized":"2023-10-03T21:21:08.825283+0000","mapping_epoch":12,"log_start":"87'201314","ondisk_log_start":"87'201314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":411,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 45s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4933001699,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":213851,"num_read_kb":833798,"num_write":209837,"num_write_kb":7903537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'32535","reported_seq":53606,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:09.343295+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:21:09.343295+0000","last_peered":"2023-10-03T21:21:09.343295+0000","last_clean":"2023-10-03T21:21:09.343295+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:21:09.343295+0000","last_undegraded":"2023-10-03T21:21:09.343295+0000","last_fullsized":"2023-10-03T21:21:09.343295+0000","mapping_epoch":15,"log_start":"84'22525","ondisk_log_start":"84'22525","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154828489,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18193,"num_read_kb":52483,"num_write":15008,"num_write_kb":49930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16007,"num_flush":4382,"num_flush_kb":17666760,"num_evict":2395,"num_evict_kb":9671989,"num_promote":2402,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'203795","reported_seq":408926,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:09.649360+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:21:09.649360+0000","last_peered":"2023-10-03T21:21:09.649360+0000","last_clean":"2023-10-03T21:21:09.649360+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:21:09.649360+0000","last_undegraded":"2023-10-03T21:21:09.649360+0000","last_fullsized":"2023-10-03T21:21:09.649360+0000","mapping_epoch":12,"log_start":"87'193709","ondisk_log_start":"87'193709","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":442,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 60s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4679172417,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":204914,"num_read_kb":799341,"num_write":202355,"num_write_kb":6930616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'32389","reported_seq":52790,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:08.463836+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:21:08.463836+0000","last_peered":"2023-10-03T21:21:08.463836+0000","last_clean":"2023-10-03T21:21:08.463836+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:21:08.463836+0000","last_undegraded":"2023-10-03T21:21:08.463836+0000","last_fullsized":"2023-10-03T21:21:08.463836+0000","mapping_epoch":15,"log_start":"84'22330","ondisk_log_start":"84'22330","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146423799,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19075,"num_read_kb":50190,"num_write":14015,"num_write_kb":45525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16174,"num_flush":3294,"num_flush_kb":13218868,"num_evict":2484,"num_evict_kb":9981584,"num_promote":2495,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"90'213689","reported_seq":429465,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:10.261235+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:21:10.261235+0000","last_peered":"2023-10-03T21:21:10.261235+0000","last_clean":"2023-10-03T21:21:10.261235+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:21:10.261235+0000","last_undegraded":"2023-10-03T21:21:10.261235+0000","last_fullsized":"2023-10-03T21:21:10.261235+0000","mapping_epoch":12,"log_start":"87'203614","ondisk_log_start":"87'203614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":388,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5060084133,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":215559,"num_read_kb":842430,"num_write":212197,"num_write_kb":7716103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'28568","reported_seq":47578,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:09.628008+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:21:09.628008+0000","last_peered":"2023-10-03T21:21:09.628008+0000","last_clean":"2023-10-03T21:21:09.628008+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:21:09.628008+0000","last_undegraded":"2023-10-03T21:21:09.628008+0000","last_fullsized":"2023-10-03T21:21:09.628008+0000","mapping_epoch":15,"log_start":"84'18530","ondisk_log_start":"84'18530","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":138150539,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16193,"num_read_kb":50252,"num_write":14338,"num_write_kb":47317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16784,"num_flush":3686,"num_flush_kb":14861936,"num_evict":2428,"num_evict_kb":9822205,"num_promote":2435,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"90'206148","reported_seq":415064,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:10.234873+0000","last_change":"2023-10-03T21:21:04.394448+0000","last_active":"2023-10-03T21:21:10.234873+0000","last_peered":"2023-10-03T21:21:10.234873+0000","last_clean":"2023-10-03T21:21:10.234873+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:21:10.234873+0000","last_undegraded":"2023-10-03T21:21:10.234873+0000","last_fullsized":"2023-10-03T21:21:10.234873+0000","mapping_epoch":12,"log_start":"86'196113","ondisk_log_start":"86'196113","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4757569987,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":208674,"num_read_kb":815018,"num_write":204984,"num_write_kb":6939869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'40253","reported_seq":65326,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:08.786512+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:21:08.786512+0000","last_peered":"2023-10-03T21:21:08.786512+0000","last_clean":"2023-10-03T21:21:08.786512+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:21:08.786512+0000","last_undegraded":"2023-10-03T21:21:08.786512+0000","last_fullsized":"2023-10-03T21:21:08.786512+0000","mapping_epoch":15,"log_start":"84'30225","ondisk_log_start":"84'30225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154787447,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":24215,"num_read_kb":55913,"num_write":14292,"num_write_kb":46782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15789,"num_flush":3619,"num_flush_kb":14502873,"num_evict":2394,"num_evict_kb":9617243,"num_promote":2404,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'211824","reported_seq":423636,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:08.804891+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:21:08.804891+0000","last_peered":"2023-10-03T21:21:08.804891+0000","last_clean":"2023-10-03T21:21:08.804891+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:21:08.804891+0000","last_undegraded":"2023-10-03T21:21:08.804891+0000","last_fullsized":"2023-10-03T21:21:08.804891+0000","mapping_epoch":12,"log_start":"86'201807","ondisk_log_start":"86'201807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4875813248,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":211594,"num_read_kb":827150,"num_write":210636,"num_write_kb":9711139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":249,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:16.661774+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:20:16.661774+0000","last_peered":"2023-10-03T21:20:16.661774+0000","last_clean":"2023-10-03T21:20:16.661774+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:20:16.661774+0000","last_undegraded":"2023-10-03T21:20:16.661774+0000","last_fullsized":"2023-10-03T21:20:16.661774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":594190274,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":77676,"num_read_kb":208838,"num_write":57653,"num_write_kb":189554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64754,"num_flush":14981,"num_flush_kb":60250437,"num_evict":9701,"num_evict_kb":39093021,"num_promote":9736,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1192787968,"data_stored":1192501124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40135,"ondisk_log_size":40135,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39331826832,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1689727,"num_read_kb":6596212,"num_write":1665506,"num_write_kb":66726303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41331601408,"data_stored":47335483680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80377,"ondisk_log_size":80377,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739182,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9070144,"kb_used_data":8106168,"kb_used_omap":376,"kb_used_meta":963591,"kb_avail":85301696,"statfs":{"total":96636764160,"available":87348936704,"internally_reserved":0,"allocated":8300716032,"data_stored":9539421705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":986718109},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.012,"15min":1.04},"min":{"1min":0.221,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":2.8330000000000002,"5min":7.907,"15min":19.114000000000001},"last":5.5670000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.90200000000000002,"15min":0.97799999999999998},"min":{"1min":0.20699999999999999,"5min":0.183,"15min":0.183},"max":{"1min":3.0030000000000001,"5min":10.976000000000001,"15min":15.244},"last":4.875}]},{"osd":1,"last update":"Tue Oct 3 21:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.0509999999999999,"15min":0.876},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.081,"5min":30.677,"15min":30.677},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.7689999999999999,"15min":1.3819999999999999},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.0430000000000001,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.28699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739173,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22163364,"kb_used_data":20476060,"kb_used_omap":961,"kb_used_meta":1686334,"kb_avail":72208476,"statfs":{"total":96636764160,"available":73941479424,"internally_reserved":0,"allocated":20967485440,"data_stored":23969123965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1726806053},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,9,13,11,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2,"15min":2.6960000000000002},"min":{"1min":0.183,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":5.9269999999999996,"5min":17.309999999999999,"15min":28.088999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.014,"15min":1.4139999999999999},"min":{"1min":0.19900000000000001,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":5.8140000000000001,"5min":5.9660000000000002,"15min":14.116},"last":0.26600000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.2889999999999999,"15min":1.278},"min":{"1min":0.20499999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":7.9539999999999997,"5min":11.577,"15min":24.614999999999998},"last":0.318},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.4159999999999999,"15min":1.2509999999999999},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":14.201000000000001,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739184,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14333404,"kb_used_data":12947860,"kb_used_omap":597,"kb_used_meta":1384938,"kb_avail":80038436,"statfs":{"total":96636764160,"available":81959358464,"internally_reserved":0,"allocated":13258608640,"data_stored":15020850299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1418177441},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.153,"15min":1.331},"min":{"1min":0.19,"5min":0.17499999999999999,"15min":0.13900000000000001},"max":{"1min":4.9050000000000002,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.276,"15min":1.0129999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":5.2670000000000003,"5min":14.673999999999999,"15min":21.204999999999998},"last":2.0630000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95799999999999996,"15min":1.145},"min":{"1min":0.22600000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.4880000000000004,"5min":8.0410000000000004,"15min":10.914999999999999},"last":2.125},{"interface":"front","average":{"1min":0.37,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.872,"5min":6.5709999999999997,"15min":6.827},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12964409344,"data_stored":14727088040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20665442304,"data_stored":23667453072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7701749760,"data_stored":8940942568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293044224,"data_stored":292974932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301268992,"data_stored":301194862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":598474752,"data_stored":598331330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:21:13.969 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.969 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.970 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.970 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.970 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.970 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.971 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.971 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.971 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.971 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.971 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:13.972 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:21:22.292 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:22.272+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:21:22.292 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:22.280+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:21:22.321 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:22.316+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:21:22.321 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:22.316+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:21:25.243 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:25.236+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:21:25.290 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:25.284+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:21:33.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:21:34.743 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:21:34.744 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:21:34.780 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2136,"stamp":"2023-10-03T21:21:34.122352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39980383688,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1775251,"num_read_kb":6835793,"num_write":1730851,"num_write_kb":67233427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64584,"num_flush":15051,"num_flush_kb":60536365,"num_evict":9747,"num_evict_kb":39280821,"num_promote":9785,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120696,"ondisk_log_size":120696,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45773136,"kb_used_data":41623860,"kb_used_omap":1935,"kb_used_meta":4147312,"kb_avail":237342384,"statfs":{"total":289910292480,"available":243038601216,"internally_reserved":0,"allocated":42622832640,"data_stored":48599216045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4246848355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,10,16,25,21,27,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45854720,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3856,"num_read_kb":15272,"num_write":3780,"num_write_kb":149680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":32,"num_flush_kb":130680,"num_evict":19,"num_evict_kb":77584,"num_promote":24,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002311"},"pg_stats":[{"pgid":"2.7","version":"90'204585","reported_seq":410298,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.323835+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:21:29.323835+0000","last_peered":"2023-10-03T21:21:29.323835+0000","last_clean":"2023-10-03T21:21:29.323835+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:21:29.323835+0000","last_undegraded":"2023-10-03T21:21:29.323835+0000","last_fullsized":"2023-10-03T21:21:29.323835+0000","mapping_epoch":12,"log_start":"87'194513","ondisk_log_start":"87'194513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4904423758,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":205523,"num_read_kb":802858,"num_write":203426,"num_write_kb":9610366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'204747","reported_seq":412737,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.329797+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:21:29.329797+0000","last_peered":"2023-10-03T21:21:29.329797+0000","last_clean":"2023-10-03T21:21:29.329797+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:21:29.329797+0000","last_undegraded":"2023-10-03T21:21:29.329797+0000","last_fullsized":"2023-10-03T21:21:29.329797+0000","mapping_epoch":12,"log_start":"87'194713","ondisk_log_start":"87'194713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4971761983,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":208166,"num_read_kb":809548,"num_write":203571,"num_write_kb":9172232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'222461","reported_seq":446817,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.306276+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:21:29.306276+0000","last_peered":"2023-10-03T21:21:29.306276+0000","last_clean":"2023-10-03T21:21:29.306276+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:21:29.306276+0000","last_undegraded":"2023-10-03T21:21:29.306276+0000","last_fullsized":"2023-10-03T21:21:29.306276+0000","mapping_epoch":12,"log_start":"87'212408","ondisk_log_start":"87'212408","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5173514743,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":224211,"num_read_kb":877009,"num_write":221215,"num_write_kb":8855381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'212382","reported_seq":427447,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:29.314058+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:21:29.314058+0000","last_peered":"2023-10-03T21:21:29.314058+0000","last_clean":"2023-10-03T21:21:29.314058+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:21:29.314058+0000","last_undegraded":"2023-10-03T21:21:29.314058+0000","last_fullsized":"2023-10-03T21:21:29.314058+0000","mapping_epoch":12,"log_start":"87'202314","ondisk_log_start":"87'202314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":574,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 66s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4937716195,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":214789,"num_read_kb":837522,"num_write":210773,"num_write_kb":7968461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'32695","reported_seq":53869,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.823017+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:21:29.823017+0000","last_peered":"2023-10-03T21:21:29.823017+0000","last_clean":"2023-10-03T21:21:29.823017+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:21:29.823017+0000","last_undegraded":"2023-10-03T21:21:29.823017+0000","last_fullsized":"2023-10-03T21:21:29.823017+0000","mapping_epoch":15,"log_start":"84'22625","ondisk_log_start":"84'22625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163241645,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":18255,"num_read_kb":52731,"num_write":15086,"num_write_kb":50200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15979,"num_flush":4408,"num_flush_kb":17772968,"num_evict":2405,"num_evict_kb":9712813,"num_promote":2414,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'204768","reported_seq":410865,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:29.827021+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:21:29.827021+0000","last_peered":"2023-10-03T21:21:29.827021+0000","last_clean":"2023-10-03T21:21:29.827021+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:21:29.827021+0000","last_undegraded":"2023-10-03T21:21:29.827021+0000","last_fullsized":"2023-10-03T21:21:29.827021+0000","mapping_epoch":12,"log_start":"87'194709","ondisk_log_start":"87'194709","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":543,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 80s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4684099905,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":205861,"num_read_kb":803081,"num_write":203258,"num_write_kb":6954584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'32539","reported_seq":53026,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:28.870001+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:21:28.870001+0000","last_peered":"2023-10-03T21:21:28.870001+0000","last_clean":"2023-10-03T21:21:28.870001+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:21:28.870001+0000","last_undegraded":"2023-10-03T21:21:28.870001+0000","last_fullsized":"2023-10-03T21:21:28.870001+0000","mapping_epoch":15,"log_start":"84'22530","ondisk_log_start":"84'22530","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154824656,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19126,"num_read_kb":50394,"num_write":14075,"num_write_kb":45719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16135,"num_flush":3310,"num_flush_kb":13284184,"num_evict":2495,"num_evict_kb":10026428,"num_promote":2508,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"90'214695","reported_seq":431455,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:30.401791+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:21:30.401791+0000","last_peered":"2023-10-03T21:21:30.401791+0000","last_clean":"2023-10-03T21:21:30.401791+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:21:30.401791+0000","last_undegraded":"2023-10-03T21:21:30.401791+0000","last_fullsized":"2023-10-03T21:21:30.401791+0000","mapping_epoch":12,"log_start":"87'204614","ondisk_log_start":"87'204614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":526,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 77s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5061669285,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":216523,"num_read_kb":846258,"num_write":213106,"num_write_kb":7756519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'28690","reported_seq":47771,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.823187+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:21:29.823187+0000","last_peered":"2023-10-03T21:21:29.823187+0000","last_clean":"2023-10-03T21:21:29.823187+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:21:29.823187+0000","last_undegraded":"2023-10-03T21:21:29.823187+0000","last_fullsized":"2023-10-03T21:21:29.823187+0000","mapping_epoch":15,"log_start":"84'18631","ondisk_log_start":"84'18631","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146375217,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16238,"num_read_kb":50432,"num_write":14398,"num_write_kb":47511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16694,"num_flush":3701,"num_flush_kb":14923248,"num_evict":2439,"num_evict_kb":9867213,"num_promote":2448,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"90'207041","reported_seq":416923,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:30.402062+0000","last_change":"2023-10-03T21:21:29.512625+0000","last_active":"2023-10-03T21:21:30.402062+0000","last_peered":"2023-10-03T21:21:30.402062+0000","last_clean":"2023-10-03T21:21:30.402062+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:21:30.402062+0000","last_undegraded":"2023-10-03T21:21:30.402062+0000","last_fullsized":"2023-10-03T21:21:30.402062+0000","mapping_epoch":12,"log_start":"87'197013","ondisk_log_start":"87'197013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4759146947,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":209619,"num_read_kb":818754,"num_write":205877,"num_write_kb":6947513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'40429","reported_seq":65594,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.278262+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:21:29.278262+0000","last_peered":"2023-10-03T21:21:29.278262+0000","last_clean":"2023-10-03T21:21:29.278262+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:21:29.278262+0000","last_undegraded":"2023-10-03T21:21:29.278262+0000","last_fullsized":"2023-10-03T21:21:29.278262+0000","mapping_epoch":15,"log_start":"84'30426","ondisk_log_start":"84'30426","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":142294634,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24262,"num_read_kb":56101,"num_write":14347,"num_write_kb":46944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15776,"num_flush":3632,"num_flush_kb":14555965,"num_evict":2408,"num_evict_kb":9674367,"num_promote":2415,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'212770","reported_seq":425568,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:21:29.324239+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:21:29.324239+0000","last_peered":"2023-10-03T21:21:29.324239+0000","last_clean":"2023-10-03T21:21:29.324239+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:21:29.324239+0000","last_undegraded":"2023-10-03T21:21:29.324239+0000","last_fullsized":"2023-10-03T21:21:29.324239+0000","mapping_epoch":12,"log_start":"87'202707","ondisk_log_start":"87'202707","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4880724352,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":212572,"num_read_kb":831014,"num_write":211582,"num_write_kb":9776103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":249,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:16.661774+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:20:16.661774+0000","last_peered":"2023-10-03T21:20:16.661774+0000","last_clean":"2023-10-03T21:20:16.661774+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:20:16.661774+0000","last_undegraded":"2023-10-03T21:20:16.661774+0000","last_fullsized":"2023-10-03T21:20:16.661774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606736152,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":77881,"num_read_kb":209658,"num_write":57906,"num_write_kb":190374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64584,"num_flush":15051,"num_flush_kb":60536365,"num_evict":9747,"num_evict_kb":39280821,"num_promote":9785,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1205334016,"data_stored":1205046832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40141,"ondisk_log_size":40141,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39373057168,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1697264,"num_read_kb":6626044,"num_write":1672808,"num_write_kb":67041159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41414651904,"data_stored":47392418080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80458,"ondisk_log_size":80458,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739186,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9100180,"kb_used_data":8122316,"kb_used_omap":376,"kb_used_meta":977479,"kb_avail":85271660,"statfs":{"total":96636764160,"available":87318179840,"internally_reserved":0,"allocated":8317251584,"data_stored":9551877471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1000939421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.012,"15min":1.04},"min":{"1min":0.221,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":2.8330000000000002,"5min":7.907,"15min":19.114000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.90200000000000002,"15min":0.97799999999999998},"min":{"1min":0.20699999999999999,"5min":0.183,"15min":0.183},"max":{"1min":3.0030000000000001,"5min":10.976000000000001,"15min":15.244},"last":0.39300000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.0509999999999999,"15min":0.876},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.081,"5min":30.677,"15min":30.677},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.7689999999999999,"15min":1.3819999999999999},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.0430000000000001,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359739177,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22276904,"kb_used_data":20509088,"kb_used_omap":961,"kb_used_meta":1766846,"kb_avail":72094936,"statfs":{"total":96636764160,"available":73825214464,"internally_reserved":0,"allocated":21001306112,"data_stored":23989775945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1809250341},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,7,13,5,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.9510000000000001,"15min":2.5800000000000001},"min":{"1min":0.20899999999999999,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":8.0429999999999993,"5min":17.309999999999999,"15min":28.088999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.96999999999999997,"15min":1.298},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":3.5259999999999998,"5min":5.8140000000000001,"15min":14.116},"last":0.23100000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":4.4489999999999998,"5min":11.577,"15min":24.614999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.45,"15min":1.2929999999999999},"min":{"1min":0.22500000000000001,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":4.6909999999999998,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.51300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739188,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14396052,"kb_used_data":12992456,"kb_used_omap":597,"kb_used_meta":1402986,"kb_avail":79975788,"statfs":{"total":96636764160,"available":81895206912,"internally_reserved":0,"allocated":13304274944,"data_stored":15057562629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1436658593},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,5,6,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:21:27 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.0740000000000001,"15min":1.3069999999999999},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.13900000000000001},"max":{"1min":4.2089999999999996,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.1699999999999999,"15min":0.98999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.182},"max":{"1min":3.4630000000000001,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.27700000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:21:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.0349999999999999,"15min":1.131},"min":{"1min":0.20100000000000001,"5min":0.187,"15min":0.187},"max":{"1min":7.5839999999999996,"5min":8.0410000000000004,"15min":10.914999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.637,"5min":0.91000000000000003,"15min":0.77900000000000003},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":11.329000000000001,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12997681152,"data_stored":14751401896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20707139584,"data_stored":23696051344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7709831168,"data_stored":8944964840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309682176,"data_stored":309612766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297177088,"data_stored":297102906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":598474752,"data_stored":598331160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:21:34.782 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.782 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.783 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.783 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.783 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.783 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.783 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.784 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.784 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.784 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.784 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:34.784 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:21:50.435 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.428+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:21:50.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.428+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:21:50.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.428+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:21:50.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.428+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:21:50.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.428+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:21:50.436 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.428+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:21:50.447 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.440+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:21:50.447 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:21:50.440+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:21:54.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:21:55.597 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:21:55.597 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:21:55.631 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2146,"stamp":"2023-10-03T21:21:54.129844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40024358282,"num_objects":14925,"num_object_clones":9472,"num_object_copies":29850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1782939,"num_read_kb":6866241,"num_write":1738438,"num_write_kb":67504324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64522,"num_flush":15112,"num_flush_kb":60785493,"num_evict":9792,"num_evict_kb":39464441,"num_promote":9834,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120738,"ondisk_log_size":120738,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":45975760,"kb_used_data":41705780,"kb_used_omap":1935,"kb_used_meta":4268016,"kb_avail":237139760,"statfs":{"total":289910292480,"available":242831114240,"internally_reserved":0,"allocated":42706718720,"data_stored":48663154175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4370449251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,8,5,22,18,20,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23101440,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":3819,"num_read_kb":15132,"num_write":3812,"num_write_kb":133524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":35,"num_flush_kb":142980,"num_evict":19,"num_evict_kb":77448,"num_promote":21,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006092"},"pg_stats":[{"pgid":"2.7","version":"91'205484","reported_seq":412127,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.880898+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:21:49.880898+0000","last_peered":"2023-10-03T21:21:49.880898+0000","last_clean":"2023-10-03T21:21:49.880898+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:21:49.880898+0000","last_undegraded":"2023-10-03T21:21:49.880898+0000","last_fullsized":"2023-10-03T21:21:49.880898+0000","mapping_epoch":12,"log_start":"87'195413","ondisk_log_start":"87'195413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4905988430,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":206448,"num_read_kb":806522,"num_write":204325,"num_write_kb":9622130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'205612","reported_seq":414545,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.887844+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:21:49.887844+0000","last_peered":"2023-10-03T21:21:49.887844+0000","last_clean":"2023-10-03T21:21:49.887844+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:21:49.887844+0000","last_undegraded":"2023-10-03T21:21:49.887844+0000","last_fullsized":"2023-10-03T21:21:49.887844+0000","mapping_epoch":12,"log_start":"87'195513","ondisk_log_start":"87'195513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4976238911,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":209104,"num_read_kb":813272,"num_write":204436,"num_write_kb":9228792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'223467","reported_seq":448809,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.888525+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:21:49.888525+0000","last_peered":"2023-10-03T21:21:49.888525+0000","last_clean":"2023-10-03T21:21:49.888525+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:21:49.888525+0000","last_undegraded":"2023-10-03T21:21:49.888525+0000","last_fullsized":"2023-10-03T21:21:49.888525+0000","mapping_epoch":12,"log_start":"87'213408","ondisk_log_start":"87'213408","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5175206391,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":225189,"num_read_kb":880885,"num_write":222221,"num_write_kb":8871613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'213474","reported_seq":429535,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:49.782564+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:21:49.782564+0000","last_peered":"2023-10-03T21:21:49.782564+0000","last_clean":"2023-10-03T21:21:49.782564+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:21:49.782564+0000","last_undegraded":"2023-10-03T21:21:49.782564+0000","last_fullsized":"2023-10-03T21:21:49.782564+0000","mapping_epoch":12,"log_start":"87'203414","ondisk_log_start":"87'203414","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":748,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 86s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4942410211,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":215761,"num_read_kb":841374,"num_write":211741,"num_write_kb":8017209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'32831","reported_seq":54078,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.930756+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:21:49.930756+0000","last_peered":"2023-10-03T21:21:49.930756+0000","last_clean":"2023-10-03T21:21:49.930756+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:21:49.930756+0000","last_undegraded":"2023-10-03T21:21:49.930756+0000","last_fullsized":"2023-10-03T21:21:49.930756+0000","mapping_epoch":15,"log_start":"84'22825","ondisk_log_start":"84'22825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159035035,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":18302,"num_read_kb":52919,"num_write":15153,"num_write_kb":50414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15961,"num_flush":4422,"num_flush_kb":17830172,"num_evict":2418,"num_evict_kb":9765909,"num_promote":2426,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"91'205724","reported_seq":412755,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:50.029879+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:21:50.029879+0000","last_peered":"2023-10-03T21:21:50.029879+0000","last_clean":"2023-10-03T21:21:50.029879+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:21:50.029879+0000","last_undegraded":"2023-10-03T21:21:50.029879+0000","last_fullsized":"2023-10-03T21:21:50.029879+0000","mapping_epoch":12,"log_start":"87'195710","ondisk_log_start":"87'195710","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":689,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 100s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4688744769,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":206768,"num_read_kb":806677,"num_write":204114,"num_write_kb":6982428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'32686","reported_seq":53267,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.593480+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:21:49.593480+0000","last_peered":"2023-10-03T21:21:49.593480+0000","last_clean":"2023-10-03T21:21:49.593480+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:21:49.593480+0000","last_undegraded":"2023-10-03T21:21:49.593480+0000","last_fullsized":"2023-10-03T21:21:49.593480+0000","mapping_epoch":15,"log_start":"84'22631","ondisk_log_start":"84'22631","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150695884,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19185,"num_read_kb":50630,"num_write":14144,"num_write_kb":45933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16131,"num_flush":3325,"num_flush_kb":13345440,"num_evict":2510,"num_evict_kb":10087696,"num_promote":2522,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'215720","reported_seq":433438,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:21:50.453006+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:21:50.453006+0000","last_peered":"2023-10-03T21:21:50.453006+0000","last_clean":"2023-10-03T21:21:50.453006+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:21:50.453006+0000","last_undegraded":"2023-10-03T21:21:50.453006+0000","last_fullsized":"2023-10-03T21:21:50.453006+0000","mapping_epoch":12,"log_start":"87'205715","ondisk_log_start":"87'205715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":691,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 97s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5066314149,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":217459,"num_read_kb":849966,"num_write":214013,"num_write_kb":7800959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'28812","reported_seq":47967,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.606426+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:21:49.606426+0000","last_peered":"2023-10-03T21:21:49.606426+0000","last_clean":"2023-10-03T21:21:49.606426+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:21:49.606426+0000","last_undegraded":"2023-10-03T21:21:49.606426+0000","last_fullsized":"2023-10-03T21:21:49.606426+0000","mapping_epoch":15,"log_start":"84'18731","ondisk_log_start":"84'18731","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":159023640,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16291,"num_read_kb":50644,"num_write":14462,"num_write_kb":47734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16669,"num_flush":3714,"num_flush_kb":14976320,"num_evict":2447,"num_evict_kb":9899825,"num_promote":2460,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'207948","reported_seq":418715,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:50.500584+0000","last_change":"2023-10-03T21:21:48.929755+0000","last_active":"2023-10-03T21:21:50.500584+0000","last_peered":"2023-10-03T21:21:50.500584+0000","last_clean":"2023-10-03T21:21:50.500584+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:21:50.500584+0000","last_undegraded":"2023-10-03T21:21:50.500584+0000","last_fullsized":"2023-10-03T21:21:50.500584+0000","mapping_epoch":12,"log_start":"87'197913","ondisk_log_start":"87'197913","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4763816387,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":210489,"num_read_kb":822190,"num_write":206784,"num_write_kb":6967461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'40606","reported_seq":65878,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.384113+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:21:49.384113+0000","last_peered":"2023-10-03T21:21:49.384113+0000","last_clean":"2023-10-03T21:21:49.384113+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:21:49.384113+0000","last_undegraded":"2023-10-03T21:21:49.384113+0000","last_fullsized":"2023-10-03T21:21:49.384113+0000","mapping_epoch":15,"log_start":"84'30526","ondisk_log_start":"84'30526","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150806107,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":24317,"num_read_kb":56317,"num_write":14413,"num_write_kb":47170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15761,"num_flush":3651,"num_flush_kb":14633561,"num_evict":2417,"num_evict_kb":9711011,"num_promote":2426,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'213683","reported_seq":427437,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:49.864370+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:21:49.864370+0000","last_peered":"2023-10-03T21:21:49.864370+0000","last_clean":"2023-10-03T21:21:49.864370+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:21:49.864370+0000","last_undegraded":"2023-10-03T21:21:49.864370+0000","last_fullsized":"2023-10-03T21:21:49.864370+0000","mapping_epoch":12,"log_start":"87'203607","ondisk_log_start":"87'203607","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4885488000,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":213520,"num_read_kb":834754,"num_write":212495,"num_write_kb":9820587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":250,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:33.081722+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:20:33.081722+0000","last_peered":"2023-10-03T21:20:33.081722+0000","last_clean":"2023-10-03T21:20:33.081722+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:20:33.081722+0000","last_undegraded":"2023-10-03T21:20:33.081722+0000","last_fullsized":"2023-10-03T21:20:33.081722+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":619560666,"num_objects":230,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":78095,"num_read_kb":210510,"num_write":58172,"num_write_kb":191251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64522,"num_flush":15112,"num_flush_kb":60785493,"num_evict":9792,"num_evict_kb":39464441,"num_promote":9834,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1235197952,"data_stored":1234910578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40222,"ondisk_log_size":40222,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39404207248,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1704738,"num_read_kb":6655640,"num_write":1680129,"num_write_kb":67311179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41477455872,"data_stored":47435155744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80419,"ondisk_log_size":80419,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739190,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152940,"kb_used_data":8144544,"kb_used_omap":376,"kb_used_meta":1008007,"kb_avail":85218900,"statfs":{"total":96636764160,"available":87264153600,"internally_reserved":0,"allocated":8340013056,"data_stored":9569809595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1032200093},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.012,"15min":1.04},"min":{"1min":0.221,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":2.8330000000000002,"5min":7.907,"15min":19.114000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.90200000000000002,"15min":0.97799999999999998},"min":{"1min":0.20699999999999999,"5min":0.183,"15min":0.183},"max":{"1min":3.0030000000000001,"5min":10.976000000000001,"15min":15.244},"last":0.23699999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:21:04 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.0509999999999999,"15min":0.876},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":3.081,"5min":30.677,"15min":30.677},"last":0.315},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.7689999999999999,"15min":1.3819999999999999},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.0430000000000001,"5min":30.045999999999999,"15min":30.045999999999999},"last":0.45000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739181,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22325500,"kb_used_data":20543796,"kb_used_omap":961,"kb_used_meta":1780734,"kb_avail":72046340,"statfs":{"total":96636764160,"available":73775452160,"internally_reserved":0,"allocated":21036847104,"data_stored":24015355344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1823471653},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,3,15,10,8,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.9510000000000001,"15min":2.5800000000000001},"min":{"1min":0.20899999999999999,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":8.0429999999999993,"5min":17.309999999999999,"15min":28.088999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.96999999999999997,"15min":1.298},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":3.5259999999999998,"5min":5.8140000000000001,"15min":14.116},"last":0.28199999999999997}]},{"osd":2,"last update":"Tue Oct 3 21:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":4.4489999999999998,"5min":11.577,"15min":24.614999999999998},"last":0.316},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.45,"15min":1.2929999999999999},"min":{"1min":0.22500000000000001,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":4.6909999999999998,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14497320,"kb_used_data":13017440,"kb_used_omap":597,"kb_used_meta":1479274,"kb_avail":79874520,"statfs":{"total":96636764160,"available":81791508480,"internally_reserved":0,"allocated":13329858560,"data_stored":15077989236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1514777505},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:21:27 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.0740000000000001,"15min":1.3069999999999999},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.13900000000000001},"max":{"1min":4.2089999999999996,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.1699999999999999,"15min":0.98999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.182},"max":{"1min":3.4630000000000001,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.35799999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:21:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.0349999999999999,"15min":1.131},"min":{"1min":0.20100000000000001,"5min":0.187,"15min":0.187},"max":{"1min":7.5839999999999996,"5min":8.0410000000000004,"15min":10.914999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.637,"5min":0.91000000000000003,"15min":0.77900000000000003},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":11.329000000000001,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13014929408,"data_stored":14763448232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20738641920,"data_stored":23717518480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7723884544,"data_stored":8954189032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322306048,"data_stored":322236595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301559808,"data_stored":301485607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":611332096,"data_stored":611188376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:21:55.632 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.633 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.633 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.633 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.633 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.633 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.634 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.634 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.634 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.634 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.634 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:21:55.635 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:21:56.022 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.016+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:21:56.023 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.016+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:21:56.043 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.036+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:21:56.054 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.040+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:21:56.055 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.040+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:21:56.055 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.040+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:21:56.055 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.040+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:21:56.060 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:56.052+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:21:57.706 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:57.700+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:21:57.707 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:57.700+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:21:57.715 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:57.708+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:21:57.715 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:57.708+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:21:57.730 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:57.724+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:21:57.730 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:57.724+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:21:59.315 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:59.308+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:21:59.327 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:59.320+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:21:59.328 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:59.320+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:21:59.328 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:21:59.320+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:22:00.217 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:00.164+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:22:00.217 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:00.164+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:22:00.217 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:00.164+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:22:00.228 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:00.224+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:22:00.248 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:00.244+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:22:12.006 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:11.999+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:22:12.818 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:12.811+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:22:12.835 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:12.827+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:22:15.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:22:16.602 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:22:16.604 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:22:16.663 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2157,"stamp":"2023-10-03T21:22:16.133910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40085998701,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1792140,"num_read_kb":6902665,"num_write":1747728,"num_write_kb":67937268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64237,"num_flush":15209,"num_flush_kb":61181965,"num_evict":9857,"num_evict_kb":39730105,"num_promote":9895,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120637,"ondisk_log_size":120637,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46199456,"kb_used_data":41821252,"kb_used_omap":1935,"kb_used_meta":4376240,"kb_avail":236916064,"statfs":{"total":289910292480,"available":242602049536,"internally_reserved":0,"allocated":42824962048,"data_stored":48742792133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4481270627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,6,21,22,26,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43532255,"num_objects":-3,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5576,"num_read_kb":22084,"num_write":5678,"num_write_kb":234840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-33,"num_flush":54,"num_flush_kb":220696,"num_evict":43,"num_evict_kb":175732,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002236"},"pg_stats":[{"pgid":"2.7","version":"91'206633","reported_seq":414344,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.454704+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:22:15.454704+0000","last_peered":"2023-10-03T21:22:15.454704+0000","last_clean":"2023-10-03T21:22:15.454704+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:22:15.454704+0000","last_undegraded":"2023-10-03T21:22:15.454704+0000","last_fullsized":"2023-10-03T21:22:15.454704+0000","mapping_epoch":12,"log_start":"87'196613","ondisk_log_start":"87'196613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4907880782,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":207511,"num_read_kb":810746,"num_write":205474,"num_write_kb":9675730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'206745","reported_seq":416799,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.475633+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:22:15.475633+0000","last_peered":"2023-10-03T21:22:15.475633+0000","last_clean":"2023-10-03T21:22:15.475633+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:22:15.475633+0000","last_undegraded":"2023-10-03T21:22:15.475633+0000","last_fullsized":"2023-10-03T21:22:15.475633+0000","mapping_epoch":12,"log_start":"87'196713","ondisk_log_start":"87'196713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4981383487,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":210222,"num_read_kb":817684,"num_write":205569,"num_write_kb":9290500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'224653","reported_seq":451261,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.461465+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:22:15.461465+0000","last_peered":"2023-10-03T21:22:15.461465+0000","last_clean":"2023-10-03T21:22:15.461465+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:22:15.461465+0000","last_undegraded":"2023-10-03T21:22:15.461465+0000","last_fullsized":"2023-10-03T21:22:15.461465+0000","mapping_epoch":12,"log_start":"88'214608","ondisk_log_start":"88'214608","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5183283703,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":226443,"num_read_kb":885829,"num_write":223407,"num_write_kb":8908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'214763","reported_seq":431955,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:15.459046+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:22:15.459046+0000","last_peered":"2023-10-03T21:22:15.459046+0000","last_clean":"2023-10-03T21:22:15.459046+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:22:15.459046+0000","last_undegraded":"2023-10-03T21:22:15.459046+0000","last_fullsized":"2023-10-03T21:22:15.459046+0000","mapping_epoch":12,"log_start":"88'204714","ondisk_log_start":"88'204714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1006,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 112s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4956979683,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":216861,"num_read_kb":845718,"num_write":212853,"num_write_kb":8091073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'33016","reported_seq":54388,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.171175+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:22:15.171175+0000","last_peered":"2023-10-03T21:22:15.171175+0000","last_clean":"2023-10-03T21:22:15.171175+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:22:15.171175+0000","last_undegraded":"2023-10-03T21:22:15.171175+0000","last_fullsized":"2023-10-03T21:22:15.171175+0000","mapping_epoch":15,"log_start":"84'22925","ondisk_log_start":"84'22925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154824260,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18381,"num_read_kb":53235,"num_write":15238,"num_write_kb":50706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15874,"num_flush":4452,"num_flush_kb":17952816,"num_evict":2429,"num_evict_kb":9810881,"num_promote":2436,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"91'206878","reported_seq":415032,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:15.365340+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:22:15.365340+0000","last_peered":"2023-10-03T21:22:15.365340+0000","last_clean":"2023-10-03T21:22:15.365340+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:22:15.365340+0000","last_undegraded":"2023-10-03T21:22:15.365340+0000","last_fullsized":"2023-10-03T21:22:15.365340+0000","mapping_epoch":12,"log_start":"87'196810","ondisk_log_start":"87'196810","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":889,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 126s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4699869505,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":207866,"num_read_kb":811017,"num_write":205123,"num_write_kb":7019176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'32882","reported_seq":53571,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.432077+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:22:15.432077+0000","last_peered":"2023-10-03T21:22:15.432077+0000","last_clean":"2023-10-03T21:22:15.432077+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:22:15.432077+0000","last_undegraded":"2023-10-03T21:22:15.432077+0000","last_fullsized":"2023-10-03T21:22:15.432077+0000","mapping_epoch":15,"log_start":"84'22831","ondisk_log_start":"84'22831","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150597503,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":19245,"num_read_kb":50870,"num_write":14230,"num_write_kb":46197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16054,"num_flush":3351,"num_flush_kb":13451708,"num_evict":2529,"num_evict_kb":10165364,"num_promote":2541,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'217009","reported_seq":435846,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:15.639238+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:22:15.639238+0000","last_peered":"2023-10-03T21:22:15.639238+0000","last_clean":"2023-10-03T21:22:15.639238+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:22:15.639238+0000","last_undegraded":"2023-10-03T21:22:15.639238+0000","last_fullsized":"2023-10-03T21:22:15.639238+0000","mapping_epoch":12,"log_start":"88'206915","ondisk_log_start":"88'206915","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":906,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5074284965,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":218555,"num_read_kb":854322,"num_write":215144,"num_write_kb":7850395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'28988","reported_seq":48232,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:14.995649+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:22:14.995649+0000","last_peered":"2023-10-03T21:22:14.995649+0000","last_clean":"2023-10-03T21:22:14.995649+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:22:14.995649+0000","last_undegraded":"2023-10-03T21:22:14.995649+0000","last_fullsized":"2023-10-03T21:22:14.995649+0000","mapping_epoch":15,"log_start":"84'18932","ondisk_log_start":"84'18932","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154849746,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16349,"num_read_kb":50876,"num_write":14559,"num_write_kb":48048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16599,"num_flush":3733,"num_flush_kb":15053904,"num_evict":2465,"num_evict_kb":9973413,"num_promote":2476,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'209029","reported_seq":420886,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.612187+0000","last_change":"2023-10-03T21:22:10.075099+0000","last_active":"2023-10-03T21:22:15.612187+0000","last_peered":"2023-10-03T21:22:15.612187+0000","last_clean":"2023-10-03T21:22:15.612187+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:22:15.612187+0000","last_undegraded":"2023-10-03T21:22:15.612187+0000","last_fullsized":"2023-10-03T21:22:15.612187+0000","mapping_epoch":12,"log_start":"87'199013","ondisk_log_start":"87'199013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4771881411,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":211565,"num_read_kb":826450,"num_write":207865,"num_write_kb":7000357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'40834","reported_seq":66220,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.406143+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:22:15.406143+0000","last_peered":"2023-10-03T21:22:15.406143+0000","last_clean":"2023-10-03T21:22:15.406143+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:22:15.406143+0000","last_undegraded":"2023-10-03T21:22:15.406143+0000","last_fullsized":"2023-10-03T21:22:15.406143+0000","mapping_epoch":15,"log_start":"84'30828","ondisk_log_start":"84'30828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":146578984,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":24370,"num_read_kb":56529,"num_write":14498,"num_write_kb":47440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15710,"num_flush":3673,"num_flush_kb":14723537,"num_evict":2434,"num_evict_kb":9780447,"num_promote":2442,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'214819","reported_seq":429729,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:15.408587+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:22:15.408587+0000","last_peered":"2023-10-03T21:22:15.408587+0000","last_clean":"2023-10-03T21:22:15.408587+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:22:15.408587+0000","last_undegraded":"2023-10-03T21:22:15.408587+0000","last_fullsized":"2023-10-03T21:22:15.408587+0000","mapping_epoch":12,"log_start":"87'204807","ondisk_log_start":"87'204807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4902994304,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":214666,"num_read_kb":839298,"num_write":213631,"num_write_kb":9906799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":250,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T21:20:33.081722+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:20:33.081722+0000","last_peered":"2023-10-03T21:20:33.081722+0000","last_clean":"2023-10-03T21:20:33.081722+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:20:33.081722+0000","last_undegraded":"2023-10-03T21:20:33.081722+0000","last_fullsized":"2023-10-03T21:20:33.081722+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606850493,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":78345,"num_read_kb":211510,"num_write":58525,"num_write_kb":192391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64237,"num_flush":15209,"num_flush_kb":61181965,"num_evict":9857,"num_evict_kb":39730105,"num_promote":9895,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1209782272,"data_stored":1209494394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39478557840,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1713689,"num_read_kb":6691064,"num_write":1689066,"num_write_kb":67742983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41627176960,"data_stored":47545739552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80336,"ondisk_log_size":80336,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739195,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9187432,"kb_used_data":8160992,"kb_used_omap":376,"kb_used_meta":1026055,"kb_avail":85184408,"statfs":{"total":96636764160,"available":87228833792,"internally_reserved":0,"allocated":8356855808,"data_stored":9577104286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1050681245},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,3,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:22:07 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.96699999999999997,"15min":1.01},"min":{"1min":0.23400000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.5670000000000002,"5min":7.907,"15min":19.114000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.78500000000000003,"15min":0.98299999999999998},"min":{"1min":0.23599999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.875,"5min":10.976000000000001,"15min":15.244},"last":0.34799999999999998}]},{"osd":1,"last update":"Tue Oct 3 21:22:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.90700000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.6040000000000001,"5min":5.9749999999999996,"15min":30.677},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.4179999999999999,"15min":1.3879999999999999},"min":{"1min":0.223,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.0629999999999997,"5min":18.285,"15min":30.045999999999999},"last":0.39300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739186,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22451448,"kb_used_data":20601772,"kb_used_omap":961,"kb_used_meta":1848702,"kb_avail":71920392,"statfs":{"total":96636764160,"available":73646481408,"internally_reserved":0,"allocated":21096214528,"data_stored":24055385424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1893070885},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,16,12,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.9510000000000001,"15min":2.5800000000000001},"min":{"1min":0.20899999999999999,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":8.0429999999999993,"5min":17.309999999999999,"15min":28.088999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.96999999999999997,"15min":1.298},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":3.5259999999999998,"5min":5.8140000000000001,"15min":14.116},"last":28.699999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:21:26 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":4.4489999999999998,"5min":11.577,"15min":24.614999999999998},"last":28.585999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.45,"15min":1.2929999999999999},"min":{"1min":0.22500000000000001,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":4.6909999999999998,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.25}]}]},{"osd":0,"up_from":8,"seq":34359739197,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14560576,"kb_used_data":13058488,"kb_used_omap":597,"kb_used_meta":1501482,"kb_avail":79811264,"statfs":{"total":96636764160,"available":81726734336,"internally_reserved":0,"allocated":13371891712,"data_stored":15110302423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1537518497},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,4,6,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:21:27 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.0740000000000001,"15min":1.3069999999999999},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.13900000000000001},"max":{"1min":4.2089999999999996,"5min":9.8450000000000006,"15min":21.391999999999999},"last":0.309},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.1699999999999999,"15min":0.98999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.182},"max":{"1min":3.4630000000000001,"5min":14.673999999999999,"15min":21.204999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:21:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.0349999999999999,"15min":1.131},"min":{"1min":0.20100000000000001,"5min":0.187,"15min":0.187},"max":{"1min":7.5839999999999996,"5min":8.0410000000000004,"15min":10.914999999999999},"last":8.8559999999999999},{"interface":"front","average":{"1min":1.637,"5min":0.91000000000000003,"15min":0.77900000000000003},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":11.329000000000001,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13055922176,"data_stored":14794434472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20813586432,"data_stored":23772867728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7757668352,"data_stored":8978437352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309743616,"data_stored":309674006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297238528,"data_stored":297164199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602800128,"data_stored":602656189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:22:16.665 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.665 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.666 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.666 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.666 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.666 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.667 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.667 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.667 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.667 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.667 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:16.668 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:22:17.081 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:17.071+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:22:17.094 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:17.087+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:22:17.095 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:17.087+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:22:17.095 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:17.087+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:22:27.367 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:27.359+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:22:27.367 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:27.359+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:22:27.370 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:27.363+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:22:27.379 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:27.371+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:22:27.379 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:27.371+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:22:27.514 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:27.503+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:22:27.523 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:27.515+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:22:27.529 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:27.523+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:22:27.530 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:27.523+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:22:28.032 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:28.019+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:22:28.033 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:28.027+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:22:28.061 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:28.055+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:22:28.064 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:28.059+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:22:36.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:22:37.467 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:22:37.468 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:22:37.511 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2167,"stamp":"2023-10-03T21:22:36.140335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40107383706,"num_objects":14918,"num_object_clones":9472,"num_object_copies":29836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14789,"num_whiteouts":11,"num_read":1799615,"num_read_kb":6932277,"num_write":1755355,"num_write_kb":68236947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64026,"num_flush":15274,"num_flush_kb":61447533,"num_evict":9910,"num_evict_kb":39946613,"num_promote":9944,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120668,"ondisk_log_size":120668,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46397984,"kb_used_data":41879692,"kb_used_omap":1935,"kb_used_meta":4512240,"kb_avail":236717536,"statfs":{"total":289910292480,"available":242398756864,"internally_reserved":0,"allocated":42884804608,"data_stored":48792840735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4620534627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,2,7,3,15,17,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2064545,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":5532,"num_read_kb":21900,"num_write":5631,"num_write_kb":254990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-161,"num_flush":58,"num_flush_kb":236984,"num_evict":47,"num_evict_kb":192020,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":-1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002058"},"pg_stats":[{"pgid":"2.7","version":"91'207547","reported_seq":416125,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.948835+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:22:35.948835+0000","last_peered":"2023-10-03T21:22:35.948835+0000","last_clean":"2023-10-03T21:22:35.948835+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:22:35.948835+0000","last_undegraded":"2023-10-03T21:22:35.948835+0000","last_fullsized":"2023-10-03T21:22:35.948835+0000","mapping_epoch":12,"log_start":"88'197513","ondisk_log_start":"88'197513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4915765582,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":208375,"num_read_kb":814182,"num_write":206388,"num_write_kb":9724306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'207695","reported_seq":418593,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.954024+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:22:35.954024+0000","last_peered":"2023-10-03T21:22:35.954024+0000","last_clean":"2023-10-03T21:22:35.954024+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:22:35.954024+0000","last_undegraded":"2023-10-03T21:22:35.954024+0000","last_fullsized":"2023-10-03T21:22:35.954024+0000","mapping_epoch":12,"log_start":"88'197613","ondisk_log_start":"88'197613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4985942335,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":211058,"num_read_kb":820996,"num_write":206519,"num_write_kb":9359600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'225631","reported_seq":453251,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.938724+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:22:35.938724+0000","last_peered":"2023-10-03T21:22:35.938724+0000","last_clean":"2023-10-03T21:22:35.938724+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:22:35.938724+0000","last_undegraded":"2023-10-03T21:22:35.938724+0000","last_fullsized":"2023-10-03T21:22:35.938724+0000","mapping_epoch":12,"log_start":"89'215608","ondisk_log_start":"89'215608","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5185110519,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":227444,"num_read_kb":889781,"num_write":224385,"num_write_kb":8921005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'215778","reported_seq":433889,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:35.901095+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:22:35.901095+0000","last_peered":"2023-10-03T21:22:35.901095+0000","last_clean":"2023-10-03T21:22:35.901095+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:22:35.901095+0000","last_undegraded":"2023-10-03T21:22:35.901095+0000","last_fullsized":"2023-10-03T21:22:35.901095+0000","mapping_epoch":12,"log_start":"89'205714","ondisk_log_start":"89'205714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1169,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 132s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4964897251,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":217757,"num_read_kb":849250,"num_write":213754,"num_write_kb":8164065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'33140","reported_seq":54609,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.119122+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:22:35.119122+0000","last_peered":"2023-10-03T21:22:35.119122+0000","last_clean":"2023-10-03T21:22:35.119122+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:22:35.119122+0000","last_undegraded":"2023-10-03T21:22:35.119122+0000","last_fullsized":"2023-10-03T21:22:35.119122+0000","mapping_epoch":15,"log_start":"84'23125","ondisk_log_start":"84'23125","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":150597163,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":18440,"num_read_kb":53471,"num_write":15295,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15849,"num_flush":4474,"num_flush_kb":18042696,"num_evict":2442,"num_evict_kb":9863973,"num_promote":2448,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"91'207836","reported_seq":416913,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:35.485511+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:22:35.485511+0000","last_peered":"2023-10-03T21:22:35.485511+0000","last_clean":"2023-10-03T21:22:35.485511+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:22:35.485511+0000","last_undegraded":"2023-10-03T21:22:35.485511+0000","last_fullsized":"2023-10-03T21:22:35.485511+0000","mapping_epoch":12,"log_start":"89'197810","ondisk_log_start":"89'197810","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":981,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 146s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4701622593,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":208771,"num_read_kb":814609,"num_write":206016,"num_write_kb":7026836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'33033","reported_seq":53794,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.844528+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:22:35.844528+0000","last_peered":"2023-10-03T21:22:35.844528+0000","last_clean":"2023-10-03T21:22:35.844528+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:22:35.844528+0000","last_undegraded":"2023-10-03T21:22:35.844528+0000","last_fullsized":"2023-10-03T21:22:35.844528+0000","mapping_epoch":15,"log_start":"84'23031","ondisk_log_start":"84'23031","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146468637,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19280,"num_read_kb":51010,"num_write":14298,"num_write_kb":46411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15956,"num_flush":3368,"num_flush_kb":13521160,"num_evict":2543,"num_evict_kb":10222540,"num_promote":2554,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'218063","reported_seq":437884,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:35.808867+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:22:35.808867+0000","last_peered":"2023-10-03T21:22:35.808867+0000","last_clean":"2023-10-03T21:22:35.808867+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:22:35.808867+0000","last_undegraded":"2023-10-03T21:22:35.808867+0000","last_fullsized":"2023-10-03T21:22:35.808867+0000","mapping_epoch":12,"log_start":"89'208015","ondisk_log_start":"89'208015","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1077,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 143s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5079048613,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":219520,"num_read_kb":858150,"num_write":216074,"num_write_kb":7878627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'29088","reported_seq":48409,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.280894+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:22:35.280894+0000","last_peered":"2023-10-03T21:22:35.280894+0000","last_clean":"2023-10-03T21:22:35.280894+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:22:35.280894+0000","last_undegraded":"2023-10-03T21:22:35.280894+0000","last_fullsized":"2023-10-03T21:22:35.280894+0000","mapping_epoch":15,"log_start":"84'19033","ondisk_log_start":"84'19033","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150634948,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16407,"num_read_kb":51108,"num_write":14616,"num_write_kb":48215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16585,"num_flush":3741,"num_flush_kb":15086588,"num_evict":2480,"num_evict_kb":10034713,"num_promote":2490,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"91'209879","reported_seq":422612,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.805715+0000","last_change":"2023-10-03T21:22:33.800358+0000","last_active":"2023-10-03T21:22:35.805715+0000","last_peered":"2023-10-03T21:22:35.805715+0000","last_clean":"2023-10-03T21:22:35.805715+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:22:35.805715+0000","last_undegraded":"2023-10-03T21:22:35.805715+0000","last_fullsized":"2023-10-03T21:22:35.805715+0000","mapping_epoch":12,"log_start":"88'199813","ondisk_log_start":"88'199813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4773482947,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":212429,"num_read_kb":829882,"num_write":208715,"num_write_kb":7011929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'41004","reported_seq":66497,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.888517+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:22:35.888517+0000","last_peered":"2023-10-03T21:22:35.888517+0000","last_clean":"2023-10-03T21:22:35.888517+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:22:35.888517+0000","last_undegraded":"2023-10-03T21:22:35.888517+0000","last_fullsized":"2023-10-03T21:22:35.888517+0000","mapping_epoch":15,"log_start":"84'30928","ondisk_log_start":"84'30928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":142396894,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":24430,"num_read_kb":56769,"num_write":14559,"num_write_kb":47636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15636,"num_flush":3691,"num_flush_kb":14797089,"num_evict":2445,"num_evict_kb":9825387,"num_promote":2452,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'215787","reported_seq":431634,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.890007+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:22:35.890007+0000","last_peered":"2023-10-03T21:22:35.890007+0000","last_clean":"2023-10-03T21:22:35.890007+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:22:35.890007+0000","last_undegraded":"2023-10-03T21:22:35.890007+0000","last_fullsized":"2023-10-03T21:22:35.890007+0000","mapping_epoch":12,"log_start":"88'205707","ondisk_log_start":"88'205707","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4910825856,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":215598,"num_read_kb":842978,"num_write":214599,"num_write_kb":9955543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":251,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:21:31.753510+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:21:31.753510+0000","last_peered":"2023-10-03T21:21:31.753510+0000","last_clean":"2023-10-03T21:21:31.753510+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:21:31.753510+0000","last_undegraded":"2023-10-03T21:21:31.753510+0000","last_fullsized":"2023-10-03T21:21:31.753510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":590097642,"num_objects":223,"num_object_clones":0,"num_object_copies":446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":11,"num_read":78557,"num_read_kb":212358,"num_write":58768,"num_write_kb":193142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":64026,"num_flush":15274,"num_flush_kb":61447533,"num_evict":9910,"num_evict_kb":39946613,"num_promote":9944,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1184645120,"data_stored":1184356820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39516695696,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1720952,"num_read_kb":6719828,"num_write":1696450,"num_write_kb":68041911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41703604224,"data_stored":47612442912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80423,"ondisk_log_size":80423,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9229300,"kb_used_data":8170924,"kb_used_omap":376,"kb_used_meta":1057991,"kb_avail":85142540,"statfs":{"total":96636764160,"available":87185960960,"internally_reserved":0,"allocated":8367026176,"data_stored":9585361611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1083383709},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:22:07 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.96699999999999997,"15min":1.01},"min":{"1min":0.23400000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.5670000000000002,"5min":7.907,"15min":19.114000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.78500000000000003,"15min":0.98299999999999998},"min":{"1min":0.23599999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.875,"5min":10.976000000000001,"15min":15.244},"last":0.75900000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:22:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.90700000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.6040000000000001,"5min":5.9749999999999996,"15min":30.677},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.4179999999999999,"15min":1.3879999999999999},"min":{"1min":0.223,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.0629999999999997,"5min":18.285,"15min":30.045999999999999},"last":0.502}]}]},{"osd":1,"up_from":8,"seq":34359739190,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22573316,"kb_used_data":20643132,"kb_used_omap":961,"kb_used_meta":1929214,"kb_avail":71798524,"statfs":{"total":96636764160,"available":73521688576,"internally_reserved":0,"allocated":21138567168,"data_stored":24092875940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1975515173},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,2,1,2,10,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:22:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.5800000000000001,"15min":2.516},"min":{"1min":0.189,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":3.7210000000000001,"5min":15.569000000000001,"15min":28.088999999999999},"last":0.255},{"interface":"front","average":{"1min":2.1589999999999998,"5min":1.2909999999999999,"15min":1.3600000000000001},"min":{"1min":0.217,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":28.699999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.216}]},{"osd":2,"last update":"Tue Oct 3 21:22:26 2023","interfaces":[{"interface":"back","average":{"1min":2.1480000000000001,"5min":1.4179999999999999,"15min":1.3160000000000001},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":28.585999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.175,"5min":1.4410000000000001,"15min":1.262},"min":{"1min":0.25,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":7.8769999999999998,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739201,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14595368,"kb_used_data":13065636,"kb_used_omap":597,"kb_used_meta":1525034,"kb_avail":79776472,"statfs":{"total":96636764160,"available":81691107328,"internally_reserved":0,"allocated":13379211264,"data_stored":15114603184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1561635745},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89200000000000002,"15min":1.298},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":3.8010000000000002,"5min":6.601,"15min":21.391999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":1.024},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.182},"max":{"1min":3.8889999999999998,"5min":11.045999999999999,"15min":21.204999999999998},"last":0.28699999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:22:31 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0920000000000001,"15min":1.155},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":8.8559999999999999,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.95299999999999996,"15min":0.82499999999999996},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":6.2800000000000002,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.74299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13079814144,"data_stored":14815360936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20851900416,"data_stored":23806315664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7771889664,"data_stored":8990766312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301301760,"data_stored":301232111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288927744,"data_stored":288853243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":594415616,"data_stored":594271466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:22:37.513 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.514 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.514 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.514 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.514 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.515 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.515 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.515 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.515 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.515 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.515 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:37.516 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:22:41.591 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:41.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:22:41.591 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:41.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:22:41.605 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:41.599+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:22:41.605 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:41.599+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:22:41.605 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:41.599+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:22:47.877 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:22:47.871+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:22:57.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:22:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:22:58.349 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:22:58.393 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2179,"stamp":"2023-10-03T21:22:58.144635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40192436929,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1806769,"num_read_kb":6960657,"num_write":1762847,"num_write_kb":68540251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63809,"num_flush":15341,"num_flush_kb":61721321,"num_evict":9954,"num_evict_kb":40126297,"num_promote":9992,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120588,"ondisk_log_size":120588,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46641052,"kb_used_data":42038340,"kb_used_omap":1935,"kb_used_meta":4600752,"kb_avail":236474468,"statfs":{"total":289910292480,"available":242149855232,"internally_reserved":0,"allocated":43047260160,"data_stored":48923125528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4711170915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,7,13,20,11,22,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41467765,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3704,"num_read_kb":14688,"num_write":3769,"num_write_kb":125258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-139,"num_flush":27,"num_flush_kb":110396,"num_evict":25,"num_evict_kb":102112,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002100"},"pg_stats":[{"pgid":"2.7","version":"92'208435","reported_seq":417875,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:56.318509+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:22:56.318509+0000","last_peered":"2023-10-03T21:22:56.318509+0000","last_clean":"2023-10-03T21:22:56.318509+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:22:56.318509+0000","last_undegraded":"2023-10-03T21:22:56.318509+0000","last_fullsized":"2023-10-03T21:22:56.318509+0000","mapping_epoch":12,"log_start":"89'198413","ondisk_log_start":"89'198413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4923433294,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":209229,"num_read_kb":817566,"num_write":207276,"num_write_kb":9789102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'208518","reported_seq":420259,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:56.311952+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:22:56.311952+0000","last_peered":"2023-10-03T21:22:56.311952+0000","last_clean":"2023-10-03T21:22:56.311952+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:22:56.311952+0000","last_undegraded":"2023-10-03T21:22:56.311952+0000","last_fullsized":"2023-10-03T21:22:56.311952+0000","mapping_epoch":12,"log_start":"89'198513","ondisk_log_start":"89'198513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4996776255,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":211897,"num_read_kb":824328,"num_write":207342,"num_write_kb":9395552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'226639","reported_seq":455186,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:56.288175+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:22:56.288175+0000","last_peered":"2023-10-03T21:22:56.288175+0000","last_clean":"2023-10-03T21:22:56.288175+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:22:56.288175+0000","last_undegraded":"2023-10-03T21:22:56.288175+0000","last_fullsized":"2023-10-03T21:22:56.288175+0000","mapping_epoch":12,"log_start":"89'216608","ondisk_log_start":"89'216608","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5186900471,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":228369,"num_read_kb":893453,"num_write":225393,"num_write_kb":8933181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'216788","reported_seq":435782,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:56.306048+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:22:56.306048+0000","last_peered":"2023-10-03T21:22:56.306048+0000","last_clean":"2023-10-03T21:22:56.306048+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:22:56.306048+0000","last_undegraded":"2023-10-03T21:22:56.306048+0000","last_fullsized":"2023-10-03T21:22:56.306048+0000","mapping_epoch":12,"log_start":"89'206714","ondisk_log_start":"89'206714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1369,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 153s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4975497699,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":218618,"num_read_kb":852666,"num_write":214617,"num_write_kb":8196041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'33252","reported_seq":54820,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:55.607101+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:22:55.607101+0000","last_peered":"2023-10-03T21:22:55.607101+0000","last_clean":"2023-10-03T21:22:55.607101+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:22:55.607101+0000","last_undegraded":"2023-10-03T21:22:55.607101+0000","last_fullsized":"2023-10-03T21:22:55.607101+0000","mapping_epoch":15,"log_start":"84'23225","ondisk_log_start":"84'23225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158957055,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":18498,"num_read_kb":53703,"num_write":15352,"num_write_kb":51086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15805,"num_flush":4499,"num_flush_kb":18144912,"num_evict":2447,"num_evict_kb":9884397,"num_promote":2455,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'208819","reported_seq":418786,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:55.635458+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:22:55.635458+0000","last_peered":"2023-10-03T21:22:55.635458+0000","last_clean":"2023-10-03T21:22:55.635458+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:22:55.635458+0000","last_undegraded":"2023-10-03T21:22:55.635458+0000","last_fullsized":"2023-10-03T21:22:55.635458+0000","mapping_epoch":12,"log_start":"89'198810","ondisk_log_start":"89'198810","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1102,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 166s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4709245249,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":209638,"num_read_kb":818041,"num_write":206915,"num_write_kb":7054972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'33176","reported_seq":54023,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:55.656527+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:22:55.656527+0000","last_peered":"2023-10-03T21:22:55.656527+0000","last_clean":"2023-10-03T21:22:55.656527+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:22:55.656527+0000","last_undegraded":"2023-10-03T21:22:55.656527+0000","last_fullsized":"2023-10-03T21:22:55.656527+0000","mapping_epoch":15,"log_start":"84'23131","ondisk_log_start":"84'23131","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154803949,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19331,"num_read_kb":51214,"num_write":14363,"num_write_kb":46625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15908,"num_flush":3382,"num_flush_kb":13578288,"num_evict":2554,"num_evict_kb":10267436,"num_promote":2567,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'219112","reported_seq":439854,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:22:55.958017+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:22:55.958017+0000","last_peered":"2023-10-03T21:22:55.958017+0000","last_clean":"2023-10-03T21:22:55.958017+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:22:55.958017+0000","last_undegraded":"2023-10-03T21:22:55.958017+0000","last_fullsized":"2023-10-03T21:22:55.958017+0000","mapping_epoch":12,"log_start":"89'209015","ondisk_log_start":"89'209015","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1231,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 163s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5089800613,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":220422,"num_read_kb":861722,"num_write":217014,"num_write_kb":7931375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'29201","reported_seq":48599,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:55.314859+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:22:55.314859+0000","last_peered":"2023-10-03T21:22:55.314859+0000","last_clean":"2023-10-03T21:22:55.314859+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:22:55.314859+0000","last_undegraded":"2023-10-03T21:22:55.314859+0000","last_fullsized":"2023-10-03T21:22:55.314859+0000","mapping_epoch":15,"log_start":"84'19133","ondisk_log_start":"84'19133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":142344566,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16457,"num_read_kb":51308,"num_write":14682,"num_write_kb":48413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16507,"num_flush":3755,"num_flush_kb":15143776,"num_evict":2496,"num_evict_kb":10099977,"num_promote":2504,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"92'210767","reported_seq":424352,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:55.944089+0000","last_change":"2023-10-03T21:22:51.165023+0000","last_active":"2023-10-03T21:22:55.944089+0000","last_peered":"2023-10-03T21:22:55.944089+0000","last_clean":"2023-10-03T21:22:55.944089+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:22:55.944089+0000","last_undegraded":"2023-10-03T21:22:55.944089+0000","last_fullsized":"2023-10-03T21:22:55.944089+0000","mapping_epoch":12,"log_start":"89'200713","ondisk_log_start":"89'200713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4784673219,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":213270,"num_read_kb":833206,"num_write":209603,"num_write_kb":7044029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'41184","reported_seq":66768,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:56.191266+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:22:56.191266+0000","last_peered":"2023-10-03T21:22:56.191266+0000","last_clean":"2023-10-03T21:22:56.191266+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:22:56.191266+0000","last_undegraded":"2023-10-03T21:22:56.191266+0000","last_fullsized":"2023-10-03T21:22:56.191266+0000","mapping_epoch":15,"log_start":"84'31129","ondisk_log_start":"84'31129","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150736303,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24475,"num_read_kb":56949,"num_write":14630,"num_write_kb":47870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15589,"num_flush":3705,"num_flush_kb":14854345,"num_evict":2457,"num_evict_kb":9874487,"num_promote":2466,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'216711","reported_seq":433423,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:56.278475+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:22:56.278475+0000","last_peered":"2023-10-03T21:22:56.278475+0000","last_clean":"2023-10-03T21:22:56.278475+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:22:56.278475+0000","last_undegraded":"2023-10-03T21:22:56.278475+0000","last_fullsized":"2023-10-03T21:22:56.278475+0000","mapping_epoch":12,"log_start":"89'206707","ondisk_log_start":"89'206707","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4918677888,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":216459,"num_read_kb":846410,"num_write":215523,"num_write_kb":10000111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":252,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T21:22:35.921937+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:22:35.921937+0000","last_peered":"2023-10-03T21:22:35.921937+0000","last_clean":"2023-10-03T21:22:35.921937+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:22:35.921937+0000","last_undegraded":"2023-10-03T21:22:35.921937+0000","last_fullsized":"2023-10-03T21:22:35.921937+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606841873,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":78761,"num_read_kb":213174,"num_write":59027,"num_write_kb":193994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63809,"num_flush":15341,"num_flush_kb":61721321,"num_evict":9954,"num_evict_kb":40126297,"num_promote":9992,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1205592064,"data_stored":1205303330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40195,"ondisk_log_size":40195,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39585004688,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1727902,"num_read_kb":6747392,"num_write":1703683,"num_write_kb":68344363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41841401856,"data_stored":47717935392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80296,"ondisk_log_size":80296,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739203,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9271684,"kb_used_data":8213308,"kb_used_omap":376,"kb_used_meta":1057991,"kb_avail":85100156,"statfs":{"total":96636764160,"available":87142559744,"internally_reserved":0,"allocated":8410427392,"data_stored":9625452944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1083383709},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,1,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:22:07 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.96699999999999997,"15min":1.01},"min":{"1min":0.23400000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.5670000000000002,"5min":7.907,"15min":19.114000000000001},"last":3.472},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.78500000000000003,"15min":0.98299999999999998},"min":{"1min":0.23599999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.875,"5min":10.976000000000001,"15min":15.244},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:22:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.90700000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.6040000000000001,"5min":5.9749999999999996,"15min":30.677},"last":0.309},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.4179999999999999,"15min":1.3879999999999999},"min":{"1min":0.223,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.0629999999999997,"5min":18.285,"15min":30.045999999999999},"last":3.3029999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739194,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22694368,"kb_used_data":20718360,"kb_used_omap":961,"kb_used_meta":1975038,"kb_avail":71677472,"statfs":{"total":96636764160,"available":73397731328,"internally_reserved":0,"allocated":21215600640,"data_stored":24153807843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":2022438949},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,10,15,9,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:22:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.5800000000000001,"15min":2.516},"min":{"1min":0.189,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":3.7210000000000001,"5min":15.569000000000001,"15min":28.088999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.1589999999999998,"5min":1.2909999999999999,"15min":1.3600000000000001},"min":{"1min":0.217,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":28.699999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.433}]},{"osd":2,"last update":"Tue Oct 3 21:22:26 2023","interfaces":[{"interface":"back","average":{"1min":2.1480000000000001,"5min":1.4179999999999999,"15min":1.3160000000000001},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":28.585999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.175,"5min":1.4410000000000001,"15min":1.262},"min":{"1min":0.25,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":7.8769999999999998,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.42099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739205,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14675000,"kb_used_data":13106672,"kb_used_omap":597,"kb_used_meta":1567722,"kb_avail":79696840,"statfs":{"total":96636764160,"available":81609564160,"internally_reserved":0,"allocated":13421232128,"data_stored":15143864741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1605348257},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,0,2,1,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89200000000000002,"15min":1.298},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":3.8010000000000002,"5min":6.601,"15min":21.391999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":1.024},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.182},"max":{"1min":3.8889999999999998,"5min":11.045999999999999,"15min":21.204999999999998},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:22:31 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0920000000000001,"15min":1.155},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":8.8559999999999999,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.95299999999999996,"15min":0.82499999999999996},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":6.2800000000000002,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13118570496,"data_stored":14841665448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20920877056,"data_stored":23859104912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7801954304,"data_stored":9017165032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301371392,"data_stored":301301621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305598464,"data_stored":305523868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":598622208,"data_stored":598477841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:22:58.395 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.395 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.396 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.396 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.396 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.396 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.396 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.397 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.397 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.397 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.397 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:22:58.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-03T21:22:59.255 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-03T21:22:59.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T21:22:59.330 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:22:59.323+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:23:00.143 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T21:23:00.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T21:23:00.964 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T21:23:01.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T21:23:01.489 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:23:01.489 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:23:01.489 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:23:01.489 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:23:01.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:23:01.491 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:23:01.491 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:01.479+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:23:01.958 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T21:23:02.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-10-03T21:23:03.049 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-10-03T21:23:03.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-03T21:23:03.826 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-03T21:23:03.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-10-03T21:23:04.763 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-10-03T21:23:04.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T21:23:05.560 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T21:23:05.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-03T21:23:06.424 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-03T21:23:06.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T21:23:07.283 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T21:23:07.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-10-03T21:23:08.205 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-10-03T21:23:08.263 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:23:10.975 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:10.966+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:23:10.997 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:10.990+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:23:17.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:17.374+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:23:17.420 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:17.410+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:23:21.986 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:21.978+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:23:21.987 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:21.978+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:23:22.006 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:21.994+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:23:22.018 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:22.006+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:23:22.018 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:22.006+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:23:28.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:23:28.870 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:23:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:23:28.927 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2194,"stamp":"2023-10-03T21:23:28.157270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40244418972,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":1817398,"num_read_kb":7002741,"num_write":1774041,"num_write_kb":69017435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63516,"num_flush":15442,"num_flush_kb":62133993,"num_evict":10026,"num_evict_kb":40420709,"num_promote":10063,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120727,"ondisk_log_size":120727,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46918956,"kb_used_data":42137236,"kb_used_omap":1935,"kb_used_meta":4779760,"kb_avail":236196564,"statfs":{"total":289910292480,"available":241865281536,"internally_reserved":0,"allocated":43148529664,"data_stored":48987575408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4894475107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,2,7,20,14,12,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8240974,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":5322,"num_read_kb":21076,"num_write":5503,"num_write_kb":221887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-178,"num_flush":42,"num_flush_kb":171648,"num_evict":32,"num_evict_kb":130856,"num_promote":28,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006855"},"pg_stats":[{"pgid":"2.7","version":"92'209770","reported_seq":420487,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.941356+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:23:26.941356+0000","last_peered":"2023-10-03T21:23:26.941356+0000","last_clean":"2023-10-03T21:23:26.941356+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:23:26.941356+0000","last_undegraded":"2023-10-03T21:23:26.941356+0000","last_fullsized":"2023-10-03T21:23:26.941356+0000","mapping_epoch":12,"log_start":"89'199713","ondisk_log_start":"89'199713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4928741710,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":210501,"num_read_kb":822622,"num_write":208611,"num_write_kb":9847462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'209843","reported_seq":422860,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.951575+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:23:26.951575+0000","last_peered":"2023-10-03T21:23:26.951575+0000","last_clean":"2023-10-03T21:23:26.951575+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:23:26.951575+0000","last_undegraded":"2023-10-03T21:23:26.951575+0000","last_fullsized":"2023-10-03T21:23:26.951575+0000","mapping_epoch":12,"log_start":"89'199813","ondisk_log_start":"89'199813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":4999045439,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":213161,"num_read_kb":829340,"num_write":208667,"num_write_kb":9425380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'228063","reported_seq":458008,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.947580+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:23:26.947580+0000","last_peered":"2023-10-03T21:23:26.947580+0000","last_clean":"2023-10-03T21:23:26.947580+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:23:26.947580+0000","last_undegraded":"2023-10-03T21:23:26.947580+0000","last_fullsized":"2023-10-03T21:23:26.947580+0000","mapping_epoch":12,"log_start":"89'218008","ondisk_log_start":"89'218008","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5201699319,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":229750,"num_read_kb":898917,"num_write":226817,"num_write_kb":9000117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'218385","reported_seq":438681,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:26.937860+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:23:26.937860+0000","last_peered":"2023-10-03T21:23:26.937860+0000","last_clean":"2023-10-03T21:23:26.937860+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:23:26.937860+0000","last_undegraded":"2023-10-03T21:23:26.937860+0000","last_fullsized":"2023-10-03T21:23:26.937860+0000","mapping_epoch":12,"log_start":"89'208314","ondisk_log_start":"89'208314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1602,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4978078179,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":219885,"num_read_kb":857658,"num_write":216050,"num_write_kb":8279349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'33416","reported_seq":55108,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:25.975068+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:23:25.975068+0000","last_peered":"2023-10-03T21:23:25.975068+0000","last_clean":"2023-10-03T21:23:25.975068+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:23:25.975068+0000","last_undegraded":"2023-10-03T21:23:25.975068+0000","last_fullsized":"2023-10-03T21:23:25.975068+0000","mapping_epoch":15,"log_start":"84'23326","ondisk_log_start":"84'23326","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158940617,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":18577,"num_read_kb":54011,"num_write":15439,"num_write_kb":51384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15751,"num_flush":4521,"num_flush_kb":18234772,"num_evict":2459,"num_evict_kb":9933481,"num_promote":2467,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'210220","reported_seq":421446,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:26.016908+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:23:26.016908+0000","last_peered":"2023-10-03T21:23:26.016908+0000","last_clean":"2023-10-03T21:23:26.016908+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:23:26.016908+0000","last_undegraded":"2023-10-03T21:23:26.016908+0000","last_fullsized":"2023-10-03T21:23:26.016908+0000","mapping_epoch":12,"log_start":"89'200210","ondisk_log_start":"89'200210","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1340,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 196s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4717736257,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":210868,"num_read_kb":822905,"num_write":208152,"num_write_kb":7121224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'33399","reported_seq":54385,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.642680+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:23:26.642680+0000","last_peered":"2023-10-03T21:23:26.642680+0000","last_clean":"2023-10-03T21:23:26.642680+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:23:26.642680+0000","last_undegraded":"2023-10-03T21:23:26.642680+0000","last_fullsized":"2023-10-03T21:23:26.642680+0000","mapping_epoch":15,"log_start":"84'23331","ondisk_log_start":"84'23331","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150593144,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":19412,"num_read_kb":51538,"num_write":14464,"num_write_kb":46951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15791,"num_flush":3410,"num_flush_kb":13692588,"num_evict":2573,"num_evict_kb":10345108,"num_promote":2585,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'220671","reported_seq":442753,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:26.359189+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:23:26.359189+0000","last_peered":"2023-10-03T21:23:26.359189+0000","last_clean":"2023-10-03T21:23:26.359189+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:23:26.359189+0000","last_undegraded":"2023-10-03T21:23:26.359189+0000","last_fullsized":"2023-10-03T21:23:26.359189+0000","mapping_epoch":12,"log_start":"89'210615","ondisk_log_start":"89'210615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1464,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 193s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5098434981,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":221726,"num_read_kb":866866,"num_write":218400,"num_write_kb":8018591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'29397","reported_seq":48921,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:25.863344+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:23:25.863344+0000","last_peered":"2023-10-03T21:23:25.863344+0000","last_clean":"2023-10-03T21:23:25.863344+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:23:25.863344+0000","last_undegraded":"2023-10-03T21:23:25.863344+0000","last_fullsized":"2023-10-03T21:23:25.863344+0000","mapping_epoch":15,"log_start":"84'19333","ondisk_log_start":"84'19333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150716741,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16536,"num_read_kb":51624,"num_write":14790,"num_write_kb":48755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16458,"num_flush":3785,"num_flush_kb":15266372,"num_evict":2518,"num_evict_kb":10189929,"num_promote":2528,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"92'212076","reported_seq":426978,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.357405+0000","last_change":"2023-10-03T21:23:21.382978+0000","last_active":"2023-10-03T21:23:26.357405+0000","last_peered":"2023-10-03T21:23:26.357405+0000","last_clean":"2023-10-03T21:23:26.357405+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:23:26.357405+0000","last_undegraded":"2023-10-03T21:23:26.357405+0000","last_fullsized":"2023-10-03T21:23:26.357405+0000","mapping_epoch":12,"log_start":"89'202013","ondisk_log_start":"89'202013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4790182339,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":214566,"num_read_kb":838350,"num_write":210912,"num_write_kb":7085877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'41443","reported_seq":67160,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.818951+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:23:26.818951+0000","last_peered":"2023-10-03T21:23:26.818951+0000","last_clean":"2023-10-03T21:23:26.818951+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:23:26.818951+0000","last_undegraded":"2023-10-03T21:23:26.818951+0000","last_fullsized":"2023-10-03T21:23:26.818951+0000","mapping_epoch":15,"log_start":"84'31430","ondisk_log_start":"84'31430","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":142310758,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24539,"num_read_kb":57205,"num_write":14730,"num_write_kb":48192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15516,"num_flush":3726,"num_flush_kb":14940261,"num_evict":2476,"num_evict_kb":9952191,"num_promote":2483,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'218060","reported_seq":436093,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:26.932199+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:23:26.932199+0000","last_peered":"2023-10-03T21:23:26.932199+0000","last_clean":"2023-10-03T21:23:26.932199+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:23:26.932199+0000","last_undegraded":"2023-10-03T21:23:26.932199+0000","last_fullsized":"2023-10-03T21:23:26.932199+0000","mapping_epoch":12,"log_start":"89'208007","ondisk_log_start":"89'208007","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4927349120,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":217771,"num_read_kb":851614,"num_write":216872,"num_write_kb":10042259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":253,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:41.763249+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:22:41.763249+0000","last_peered":"2023-10-03T21:22:41.763249+0000","last_clean":"2023-10-03T21:22:41.763249+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:22:41.763249+0000","last_undegraded":"2023-10-03T21:22:41.763249+0000","last_fullsized":"2023-10-03T21:22:41.763249+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602561260,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":79064,"num_read_kb":214378,"num_write":59423,"num_write_kb":195282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63516,"num_flush":15442,"num_flush_kb":62133993,"num_evict":10026,"num_evict_kb":40420709,"num_promote":10063,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1205387264,"data_stored":1205097944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40235,"ondisk_log_size":40235,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39641267344,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1738228,"num_read_kb":6788272,"num_write":1714481,"num_write_kb":68820259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41947742208,"data_stored":47787841824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80395,"ondisk_log_size":80395,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9320868,"kb_used_data":8226396,"kb_used_omap":376,"kb_used_meta":1094087,"kb_avail":85050972,"statfs":{"total":96636764160,"available":87092195328,"internally_reserved":0,"allocated":8423829504,"data_stored":9624383548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1120346013},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80600000000000005,"15min":0.99399999999999999},"min":{"1min":0.19700000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":3.472,"5min":7.907,"15min":19.114000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.85899999999999999,"15min":0.95499999999999996},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":3.9809999999999999,"5min":10.976000000000001,"15min":15.244},"last":0.26500000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.83699999999999997,"15min":0.91400000000000003},"min":{"1min":0.246,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":4.1189999999999998,"5min":5.9749999999999996,"15min":30.677},"last":0.371},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.339,"15min":1.3049999999999999},"min":{"1min":0.22900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3029999999999999,"5min":18.285,"15min":30.045999999999999},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739200,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22839924,"kb_used_data":20769580,"kb_used_omap":961,"kb_used_meta":2069374,"kb_avail":71531916,"statfs":{"total":96636764160,"available":73248681984,"internally_reserved":0,"allocated":21268049920,"data_stored":24187640566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":2119039013},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,3,14,5,12,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:22:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.5800000000000001,"15min":2.516},"min":{"1min":0.189,"5min":0.17499999999999999,"15min":0.13800000000000001},"max":{"1min":3.7210000000000001,"5min":15.569000000000001,"15min":28.088999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.1589999999999998,"5min":1.2909999999999999,"15min":1.3600000000000001},"min":{"1min":0.217,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":28.699999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Tue Oct 3 21:22:26 2023","interfaces":[{"interface":"back","average":{"1min":2.1480000000000001,"5min":1.4179999999999999,"15min":1.3160000000000001},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.19},"max":{"1min":28.585999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.4410000000000001,"15min":1.262},"min":{"1min":0.25,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":7.8769999999999998,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.40799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14758164,"kb_used_data":13141260,"kb_used_omap":597,"kb_used_meta":1616298,"kb_avail":79613676,"statfs":{"total":96636764160,"available":81524404224,"internally_reserved":0,"allocated":13456650240,"data_stored":15175551294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1655090081},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,5,8,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:22:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89200000000000002,"15min":1.298},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":3.8010000000000002,"5min":6.601,"15min":21.391999999999999},"last":1.524},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":1.024},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.182},"max":{"1min":3.8889999999999998,"5min":11.045999999999999,"15min":21.204999999999998},"last":1.653}]},{"osd":2,"last update":"Tue Oct 3 21:22:31 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0920000000000001,"15min":1.155},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":8.8559999999999999,"5min":8.8559999999999999,"15min":10.914999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.95299999999999996,"15min":0.82499999999999996},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":6.2800000000000002,"5min":11.329000000000001,"15min":11.329000000000001},"last":1.482}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13145776128,"data_stored":14865135528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20977184768,"data_stored":23896878224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7824781312,"data_stored":9025828072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309719040,"data_stored":309649166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292899806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602693632,"data_stored":602548972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:23:28.929 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.929 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.929 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.930 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.930 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.930 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.930 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.930 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.931 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.931 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.931 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:28.931 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:23:32.619 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:32.610+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:23:32.619 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:32.610+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:23:34.759 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:34.750+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:23:34.991 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:34.982+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:23:35.009 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:34.994+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:23:35.017 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:35.010+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:23:35.017 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:35.010+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:23:42.209 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:42.198+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:23:42.372 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:42.362+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:23:42.391 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:23:42.382+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:23:48.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:23:49.688 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:23:49.689 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:23:49.802 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2204,"stamp":"2023-10-03T21:23:48.174334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40281332012,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1824340,"num_read_kb":7030217,"num_write":1781312,"num_write_kb":69348244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63404,"num_flush":15512,"num_flush_kb":62419953,"num_evict":10076,"num_evict_kb":40624933,"num_promote":10113,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120675,"ondisk_log_size":120675,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46498852,"kb_used_data":42210696,"kb_used_omap":1935,"kb_used_meta":4286192,"kb_avail":236616668,"statfs":{"total":289910292480,"available":242295468032,"internally_reserved":0,"allocated":43223752704,"data_stored":49036026768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4389061475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,11,13,22,20,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24248209,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5088,"num_read_kb":20136,"num_write":5251,"num_write_kb":253443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-111,"num_flush":38,"num_flush_kb":155220,"num_evict":31,"num_evict_kb":126596,"num_promote":30,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011733"},"pg_stats":[{"pgid":"2.7","version":"92'210667","reported_seq":422211,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:47.467284+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:23:47.467284+0000","last_peered":"2023-10-03T21:23:47.467284+0000","last_clean":"2023-10-03T21:23:47.467284+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:23:47.467284+0000","last_undegraded":"2023-10-03T21:23:47.467284+0000","last_fullsized":"2023-10-03T21:23:47.467284+0000","mapping_epoch":12,"log_start":"89'200613","ondisk_log_start":"89'200613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4939518286,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":211327,"num_read_kb":825870,"num_write":209508,"num_write_kb":9932558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'210727","reported_seq":424579,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:47.507465+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:23:47.507465+0000","last_peered":"2023-10-03T21:23:47.507465+0000","last_clean":"2023-10-03T21:23:47.507465+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:23:47.507465+0000","last_undegraded":"2023-10-03T21:23:47.507465+0000","last_fullsized":"2023-10-03T21:23:47.507465+0000","mapping_epoch":12,"log_start":"89'200713","ondisk_log_start":"89'200713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5003551039,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":213991,"num_read_kb":832644,"num_write":209551,"num_write_kb":9461552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'228964","reported_seq":459849,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:47.468466+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:23:47.468466+0000","last_peered":"2023-10-03T21:23:47.468466+0000","last_clean":"2023-10-03T21:23:47.468466+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:23:47.468466+0000","last_undegraded":"2023-10-03T21:23:47.468466+0000","last_fullsized":"2023-10-03T21:23:47.468466+0000","mapping_epoch":12,"log_start":"89'218908","ondisk_log_start":"89'218908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5206139383,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":230681,"num_read_kb":902585,"num_write":227718,"num_write_kb":9032269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'219377","reported_seq":440539,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:47.481863+0000","last_change":"2023-10-03T21:20:23.198832+0000","last_active":"2023-10-03T21:23:47.481863+0000","last_peered":"2023-10-03T21:23:47.481863+0000","last_clean":"2023-10-03T21:23:47.481863+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:23:47.481863+0000","last_undegraded":"2023-10-03T21:23:47.481863+0000","last_fullsized":"2023-10-03T21:23:47.481863+0000","mapping_epoch":12,"log_start":"90'209314","ondisk_log_start":"90'209314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1797,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 204s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4982632931,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":220731,"num_read_kb":861018,"num_write":216906,"num_write_kb":8286857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'33568","reported_seq":55349,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:45.973173+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:23:45.973173+0000","last_peered":"2023-10-03T21:23:45.973173+0000","last_clean":"2023-10-03T21:23:45.973173+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:23:45.973173+0000","last_undegraded":"2023-10-03T21:23:45.973173+0000","last_fullsized":"2023-10-03T21:23:45.973173+0000","mapping_epoch":15,"log_start":"84'23527","ondisk_log_start":"84'23527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163220936,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18624,"num_read_kb":54199,"num_write":15524,"num_write_kb":51666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15750,"num_flush":4548,"num_flush_kb":18345000,"num_evict":2473,"num_evict_kb":9990637,"num_promote":2482,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'211151","reported_seq":423152,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:46.156470+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:23:46.156470+0000","last_peered":"2023-10-03T21:23:46.156470+0000","last_clean":"2023-10-03T21:23:46.156470+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:23:46.156470+0000","last_undegraded":"2023-10-03T21:23:46.156470+0000","last_fullsized":"2023-10-03T21:23:46.156470+0000","mapping_epoch":12,"log_start":"89'201111","ondisk_log_start":"89'201111","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1475,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 217s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4722368833,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":211624,"num_read_kb":825905,"num_write":208988,"num_write_kb":7157236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'33538","reported_seq":54603,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:47.380371+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:23:47.380371+0000","last_peered":"2023-10-03T21:23:47.380371+0000","last_clean":"2023-10-03T21:23:47.380371+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:23:47.380371+0000","last_undegraded":"2023-10-03T21:23:47.380371+0000","last_fullsized":"2023-10-03T21:23:47.380371+0000","mapping_epoch":15,"log_start":"84'23531","ondisk_log_start":"84'23531","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146431566,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19464,"num_read_kb":51746,"num_write":14511,"num_write_kb":47073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15749,"num_flush":3418,"num_flush_kb":13725296,"num_evict":2589,"num_evict_kb":10410432,"num_promote":2600,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'221710","reported_seq":444652,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:46.722403+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:23:46.722403+0000","last_peered":"2023-10-03T21:23:46.722403+0000","last_clean":"2023-10-03T21:23:46.722403+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:23:46.722403+0000","last_undegraded":"2023-10-03T21:23:46.722403+0000","last_fullsized":"2023-10-03T21:23:46.722403+0000","mapping_epoch":12,"log_start":"90'211615","ondisk_log_start":"90'211615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1621,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5100077477,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":222588,"num_read_kb":870282,"num_write":219326,"num_write_kb":8063143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'29511","reported_seq":49126,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:45.534630+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:23:45.534630+0000","last_peered":"2023-10-03T21:23:45.534630+0000","last_clean":"2023-10-03T21:23:45.534630+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:23:45.534630+0000","last_undegraded":"2023-10-03T21:23:45.534630+0000","last_fullsized":"2023-10-03T21:23:45.534630+0000","mapping_epoch":15,"log_start":"84'19433","ondisk_log_start":"84'19433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":142405945,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16597,"num_read_kb":51868,"num_write":14854,"num_write_kb":48954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16446,"num_flush":3804,"num_flush_kb":15344028,"num_evict":2532,"num_evict_kb":10247141,"num_promote":2540,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"92'212876","reported_seq":428620,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:23:46.694831+0000","last_change":"2023-10-03T21:23:46.736737+0000","last_active":"2023-10-03T21:23:46.694831+0000","last_peered":"2023-10-03T21:23:46.694831+0000","last_clean":"2023-10-03T21:23:46.694831+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:23:46.694831+0000","last_undegraded":"2023-10-03T21:23:46.694831+0000","last_fullsized":"2023-10-03T21:23:46.694831+0000","mapping_epoch":12,"log_start":"89'202813","ondisk_log_start":"89'202813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4791763395,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":215395,"num_read_kb":841622,"num_write":211712,"num_write_kb":7117669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'41636","reported_seq":67448,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:47.341178+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:23:47.341178+0000","last_peered":"2023-10-03T21:23:47.341178+0000","last_clean":"2023-10-03T21:23:47.341178+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:23:47.341178+0000","last_undegraded":"2023-10-03T21:23:47.341178+0000","last_fullsized":"2023-10-03T21:23:47.341178+0000","mapping_epoch":15,"log_start":"84'31630","ondisk_log_start":"84'31630","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150637869,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24586,"num_read_kb":57393,"num_write":14797,"num_write_kb":48434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15459,"num_flush":3742,"num_flush_kb":15005629,"num_evict":2482,"num_evict_kb":9976723,"num_promote":2491,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'218968","reported_seq":437863,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:47.457902+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:23:47.457902+0000","last_peered":"2023-10-03T21:23:47.457902+0000","last_clean":"2023-10-03T21:23:47.457902+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:23:47.457902+0000","last_undegraded":"2023-10-03T21:23:47.457902+0000","last_fullsized":"2023-10-03T21:23:47.457902+0000","mapping_epoch":12,"log_start":"89'208907","ondisk_log_start":"89'208907","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4931993984,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":218626,"num_read_kb":854994,"num_write":217780,"num_write_kb":10098939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":253,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:22:41.763249+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:22:41.763249+0000","last_peered":"2023-10-03T21:22:41.763249+0000","last_clean":"2023-10-03T21:22:41.763249+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:22:41.763249+0000","last_undegraded":"2023-10-03T21:22:41.763249+0000","last_fullsized":"2023-10-03T21:22:41.763249+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602696316,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":79271,"num_read_kb":215206,"num_write":59686,"num_write_kb":196127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63404,"num_flush":15512,"num_flush_kb":62419953,"num_evict":10076,"num_evict_kb":40624933,"num_promote":10113,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1201463296,"data_stored":1201173752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40132,"ondisk_log_size":40132,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39678045328,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1744963,"num_read_kb":6814920,"num_write":1721489,"num_write_kb":69150223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42027511808,"data_stored":47841237280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80446,"ondisk_log_size":80446,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739213,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9346912,"kb_used_data":8234392,"kb_used_omap":376,"kb_used_meta":1112135,"kb_avail":85024928,"statfs":{"total":96636764160,"available":87065526272,"internally_reserved":0,"allocated":8432017408,"data_stored":9631317964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1138827165},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80600000000000005,"15min":0.99399999999999999},"min":{"1min":0.19700000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":3.472,"5min":7.907,"15min":19.114000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.85899999999999999,"15min":0.95499999999999996},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":3.9809999999999999,"5min":10.976000000000001,"15min":15.244},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.83699999999999997,"15min":0.91400000000000003},"min":{"1min":0.246,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":4.1189999999999998,"5min":5.9749999999999996,"15min":30.677},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.339,"15min":1.3049999999999999},"min":{"1min":0.22900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3029999999999999,"5min":18.285,"15min":30.045999999999999},"last":3.581}]}]},{"osd":1,"up_from":8,"seq":34359739204,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22335652,"kb_used_data":20813020,"kb_used_omap":961,"kb_used_meta":1521662,"kb_avail":72036188,"statfs":{"total":96636764160,"available":73765056512,"internally_reserved":0,"allocated":21312532480,"data_stored":24218872467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1558181925},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,9,6,15,9,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.4770000000000001,"15min":2.2170000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.13800000000000001},"max":{"1min":6.5599999999999996,"5min":15.569000000000001,"15min":28.088999999999999},"last":4.2649999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.232,"15min":1.3129999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":4.7430000000000003,"5min":28.699999999999999,"15min":28.699999999999999},"last":2.6789999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:23:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.369,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.0090000000000003,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.251},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.278,"15min":1.202},"min":{"1min":0.21099999999999999,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":6.2370000000000001,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359739216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14816288,"kb_used_data":13163284,"kb_used_omap":597,"kb_used_meta":1652394,"kb_avail":79555552,"statfs":{"total":96636764160,"available":81464885248,"internally_reserved":0,"allocated":13479202816,"data_stored":15185836337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1692052385},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:23:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.89600000000000002,"15min":1.2889999999999999},"min":{"1min":0.27100000000000002,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":3.407,"5min":6.601,"15min":21.391999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.083,"5min":0.95299999999999996,"15min":1.046},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":6.2990000000000004,"5min":11.045999999999999,"15min":21.204999999999998},"last":0.54700000000000004}]},{"osd":2,"last update":"Tue Oct 3 21:23:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0489999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":6.2320000000000002,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.91300000000000003,"15min":0.85199999999999998},"min":{"1min":0.25900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.5459999999999998,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.78500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13180751872,"data_stored":14887819176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21014093824,"data_stored":23920909456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7832666112,"data_stored":9032508648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305688576,"data_stored":305618689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297140224,"data_stored":297065339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":598634496,"data_stored":598489724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:23:49.803 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.804 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.804 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.804 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.804 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.805 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.805 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.805 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.805 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.805 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.806 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:23:49.806 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:23:51.818 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:23:51.814+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:24:08.298 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:08.293+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:24:09.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:24:10.719 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:24:10.719 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:24:10.809 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2216,"stamp":"2023-10-03T21:24:10.178100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40352336050,"num_objects":14924,"num_object_clones":9472,"num_object_copies":29848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14790,"num_whiteouts":11,"num_read":1831744,"num_read_kb":7059525,"num_write":1788565,"num_write_kb":69695532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63282,"num_flush":15599,"num_flush_kb":62775429,"num_evict":10120,"num_evict_kb":40804709,"num_promote":10160,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7560,"num_omap_keys":149,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120690,"ondisk_log_size":120690,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46794568,"kb_used_data":42356396,"kb_used_omap":1935,"kb_used_meta":4432112,"kb_avail":236320952,"statfs":{"total":289910292480,"available":241992654848,"internally_reserved":0,"allocated":43372949504,"data_stored":49160675432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4538483555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1,11,22,13,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38899687,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":3533,"num_read_kb":13972,"num_write":3576,"num_write_kb":218330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-25,"num_flush":48,"num_flush_kb":196116,"num_evict":24,"num_evict_kb":98076,"num_promote":23,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":-2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002022"},"pg_stats":[{"pgid":"2.7","version":"93'211494","reported_seq":423866,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.908855+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:24:07.908855+0000","last_peered":"2023-10-03T21:24:07.908855+0000","last_clean":"2023-10-03T21:24:07.908855+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:24:07.908855+0000","last_undegraded":"2023-10-03T21:24:07.908855+0000","last_fullsized":"2023-10-03T21:24:07.908855+0000","mapping_epoch":12,"log_start":"89'201413","ondisk_log_start":"89'201413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4946895182,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":212142,"num_read_kb":829102,"num_write":210335,"num_write_kb":9997102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'211618","reported_seq":426334,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.921544+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:24:07.921544+0000","last_peered":"2023-10-03T21:24:07.921544+0000","last_clean":"2023-10-03T21:24:07.921544+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:24:07.921544+0000","last_undegraded":"2023-10-03T21:24:07.921544+0000","last_fullsized":"2023-10-03T21:24:07.921544+0000","mapping_epoch":12,"log_start":"89'201613","ondisk_log_start":"89'201613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5008228671,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":214847,"num_read_kb":836028,"num_write":210442,"num_write_kb":9493696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'229914","reported_seq":461786,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.875553+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:24:07.875553+0000","last_peered":"2023-10-03T21:24:07.875553+0000","last_clean":"2023-10-03T21:24:07.875553+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:24:07.875553+0000","last_undegraded":"2023-10-03T21:24:07.875553+0000","last_fullsized":"2023-10-03T21:24:07.875553+0000","mapping_epoch":12,"log_start":"90'219908","ondisk_log_start":"90'219908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5214142967,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":231659,"num_read_kb":906457,"num_write":228668,"num_write_kb":9060597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'220316","reported_seq":442437,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.849685+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:24:07.849685+0000","last_peered":"2023-10-03T21:24:07.849685+0000","last_clean":"2023-10-03T21:24:07.849685+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:24:07.849685+0000","last_undegraded":"2023-10-03T21:24:07.849685+0000","last_fullsized":"2023-10-03T21:24:07.849685+0000","mapping_epoch":12,"log_start":"90'210314","ondisk_log_start":"90'210314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":4993241571,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":221666,"num_read_kb":864722,"num_write":217786,"num_write_kb":8339337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'33724","reported_seq":55589,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:06.200681+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:24:06.200681+0000","last_peered":"2023-10-03T21:24:06.200681+0000","last_clean":"2023-10-03T21:24:06.200681+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:24:06.200681+0000","last_undegraded":"2023-10-03T21:24:06.200681+0000","last_fullsized":"2023-10-03T21:24:06.200681+0000","mapping_epoch":15,"log_start":"84'23627","ondisk_log_start":"84'23627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163126701,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":18664,"num_read_kb":54359,"num_write":15585,"num_write_kb":51868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15723,"num_flush":4576,"num_flush_kb":18459292,"num_evict":2483,"num_evict_kb":10031545,"num_promote":2492,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'212088","reported_seq":425001,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:06.375694+0000","last_change":"2023-10-03T21:20:09.072979+0000","last_active":"2023-10-03T21:24:06.375694+0000","last_peered":"2023-10-03T21:24:06.375694+0000","last_clean":"2023-10-03T21:24:06.375694+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:24:06.375694+0000","last_undegraded":"2023-10-03T21:24:06.375694+0000","last_fullsized":"2023-10-03T21:24:06.375694+0000","mapping_epoch":12,"log_start":"90'202011","ondisk_log_start":"90'202011","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1672,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 237s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4727001409,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":212519,"num_read_kb":829465,"num_write":209788,"num_write_kb":7201332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'33675","reported_seq":54827,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.705161+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:24:07.705161+0000","last_peered":"2023-10-03T21:24:07.705161+0000","last_clean":"2023-10-03T21:24:07.705161+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:24:07.705161+0000","last_undegraded":"2023-10-03T21:24:07.705161+0000","last_fullsized":"2023-10-03T21:24:07.705161+0000","mapping_epoch":15,"log_start":"84'23631","ondisk_log_start":"84'23631","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154783286,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19507,"num_read_kb":51918,"num_write":14581,"num_write_kb":47303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15725,"num_flush":3444,"num_flush_kb":13831500,"num_evict":2601,"num_evict_kb":10459380,"num_promote":2614,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'222712","reported_seq":446600,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:06.903903+0000","last_change":"2023-10-03T21:20:12.745647+0000","last_active":"2023-10-03T21:24:06.903903+0000","last_peered":"2023-10-03T21:24:06.903903+0000","last_clean":"2023-10-03T21:24:06.903903+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:24:06.903903+0000","last_undegraded":"2023-10-03T21:24:06.903903+0000","last_fullsized":"2023-10-03T21:24:06.903903+0000","mapping_epoch":12,"log_start":"90'212615","ondisk_log_start":"90'212615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1771,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 234s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5107720613,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":223493,"num_read_kb":873850,"num_write":220225,"num_write_kb":8103511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'29646","reported_seq":49334,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:05.886372+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:24:05.886372+0000","last_peered":"2023-10-03T21:24:05.886372+0000","last_clean":"2023-10-03T21:24:05.886372+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:24:05.886372+0000","last_undegraded":"2023-10-03T21:24:05.886372+0000","last_fullsized":"2023-10-03T21:24:05.886372+0000","mapping_epoch":15,"log_start":"84'19634","ondisk_log_start":"84'19634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150733088,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16642,"num_read_kb":52048,"num_write":14919,"num_write_kb":49168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16421,"num_flush":3820,"num_flush_kb":15409464,"num_evict":2543,"num_evict_kb":10292141,"num_promote":2553,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'213806","reported_seq":430474,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:06.890494+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:24:06.890494+0000","last_peered":"2023-10-03T21:24:06.890494+0000","last_clean":"2023-10-03T21:24:06.890494+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:24:06.890494+0000","last_undegraded":"2023-10-03T21:24:06.890494+0000","last_fullsized":"2023-10-03T21:24:06.890494+0000","mapping_epoch":12,"log_start":"89'203713","ondisk_log_start":"89'203713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":84,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 9s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4805763523,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":216296,"num_read_kb":845178,"num_write":212584,"num_write_kb":7170137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'41768","reported_seq":67675,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.731768+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:24:07.731768+0000","last_peered":"2023-10-03T21:24:07.731768+0000","last_clean":"2023-10-03T21:24:07.731768+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:24:07.731768+0000","last_undegraded":"2023-10-03T21:24:07.731768+0000","last_fullsized":"2023-10-03T21:24:07.731768+0000","mapping_epoch":15,"log_start":"84'31730","ondisk_log_start":"84'31730","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":146492671,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":24632,"num_read_kb":57577,"num_write":14855,"num_write_kb":48620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15413,"num_flush":3759,"num_flush_kb":15075173,"num_evict":2493,"num_evict_kb":10021643,"num_promote":2501,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'219848","reported_seq":439697,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:07.904566+0000","last_change":"2023-10-03T20:19:04.566898+0000","last_active":"2023-10-03T21:24:07.904566+0000","last_peered":"2023-10-03T21:24:07.904566+0000","last_clean":"2023-10-03T21:24:07.904566+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:24:07.904566+0000","last_undegraded":"2023-10-03T21:24:07.904566+0000","last_fullsized":"2023-10-03T21:24:07.904566+0000","mapping_epoch":12,"log_start":"90'209807","ondisk_log_start":"90'209807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4933616000,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":219571,"num_read_kb":858730,"num_write":218660,"num_write_kb":10130967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":254,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:08.205421+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:23:08.205421+0000","last_peered":"2023-10-03T21:23:08.205421+0000","last_clean":"2023-10-03T21:23:08.205421+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:23:08.205421+0000","last_undegraded":"2023-10-03T21:23:08.205421+0000","last_fullsized":"2023-10-03T21:23:08.205421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":615135746,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":11,"num_read":79445,"num_read_kb":215902,"num_write":59940,"num_write_kb":196959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63282,"num_flush":15599,"num_flush_kb":62775429,"num_evict":10120,"num_evict_kb":40804709,"num_promote":10160,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1230561280,"data_stored":1230271492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39736609936,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1752193,"num_read_kb":6843532,"num_write":1728488,"num_write_kb":69496679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42142543872,"data_stored":47928731936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80402,"ondisk_log_size":80402,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739217,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9400972,"kb_used_data":8270472,"kb_used_omap":376,"kb_used_meta":1126023,"kb_avail":84970868,"statfs":{"total":96636764160,"available":87010168832,"internally_reserved":0,"allocated":8468963328,"data_stored":9665629846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1153048477},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,1,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80600000000000005,"15min":0.99399999999999999},"min":{"1min":0.19700000000000001,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":3.472,"5min":7.907,"15min":19.114000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.85899999999999999,"15min":0.95499999999999996},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":3.9809999999999999,"5min":10.976000000000001,"15min":15.244},"last":0.435}]},{"osd":1,"last update":"Tue Oct 3 21:23:11 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.83699999999999997,"15min":0.91400000000000003},"min":{"1min":0.246,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":4.1189999999999998,"5min":5.9749999999999996,"15min":30.677},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.339,"15min":1.3049999999999999},"min":{"1min":0.22900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3029999999999999,"5min":18.285,"15min":30.045999999999999},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739208,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22487076,"kb_used_data":20874708,"kb_used_omap":961,"kb_used_meta":1611390,"kb_avail":71884764,"statfs":{"total":96636764160,"available":73609998336,"internally_reserved":0,"allocated":21375700992,"data_stored":24266766737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1650063397},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,8,15,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.4770000000000001,"15min":2.2170000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.13800000000000001},"max":{"1min":6.5599999999999996,"5min":15.569000000000001,"15min":28.088999999999999},"last":0.441},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.232,"15min":1.3129999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":4.7430000000000003,"5min":28.699999999999999,"15min":28.699999999999999},"last":3.0419999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:23:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.369,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.0090000000000003,"5min":28.585999999999999,"15min":28.585999999999999},"last":3.1869999999999998},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.278,"15min":1.202},"min":{"1min":0.21099999999999999,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":6.2370000000000001,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.215}]}]},{"osd":0,"up_from":8,"seq":34359739220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14906520,"kb_used_data":13211216,"kb_used_omap":597,"kb_used_meta":1694698,"kb_avail":79465320,"statfs":{"total":96636764160,"available":81372487680,"internally_reserved":0,"allocated":13528285184,"data_stored":15228278849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1735371681},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:23:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.89600000000000002,"15min":1.2889999999999999},"min":{"1min":0.27100000000000002,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":3.407,"5min":6.601,"15min":21.391999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.083,"5min":0.95299999999999996,"15min":1.046},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":6.2990000000000004,"5min":11.045999999999999,"15min":21.204999999999998},"last":0.432}]},{"osd":2,"last update":"Tue Oct 3 21:23:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0489999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":6.2320000000000002,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.247},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.91300000000000003,"15min":0.85199999999999998},"min":{"1min":0.25900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.5459999999999998,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13213233152,"data_stored":14913664936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21073264640,"data_stored":23964761232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7856046080,"data_stored":9050305768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313925632,"data_stored":313855693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301346816,"data_stored":301271861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":615288832,"data_stored":615143938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:24:10.811 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.811 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.811 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.812 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.812 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.812 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.813 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.813 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.813 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.813 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:10.814 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:24:12.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:12.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:24:12.750 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:12.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:24:12.751 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:12.745+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:24:12.766 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:12.761+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:24:12.766 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:12.761+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:24:12.766 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:12.761+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:24:30.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:24:31.564 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:24:31.565 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:24:31.628 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2226,"stamp":"2023-10-03T21:24:30.182941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40387414196,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1839464,"num_read_kb":7090065,"num_write":1796281,"num_write_kb":69979141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63284,"num_flush":15665,"num_flush_kb":63045021,"num_evict":10174,"num_evict_kb":41025281,"num_promote":10212,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15011,"num_omap_keys":266,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120588,"ondisk_log_size":120588,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":46894876,"kb_used_data":42415040,"kb_used_omap":1935,"kb_used_meta":4477872,"kb_avail":236220644,"statfs":{"total":289910292480,"available":241889939456,"internally_reserved":0,"allocated":43433000960,"data_stored":49193508972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4585341795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,6,13,19,26,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31023121,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5080,"num_read_kb":20080,"num_write":5134,"num_write_kb":191739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":17,"num_flush":42,"num_flush_kb":171504,"num_evict":30,"num_evict_kb":122444,"num_promote":31,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7451,"num_omap_keys":117,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003382"},"pg_stats":[{"pgid":"2.7","version":"93'212402","reported_seq":425699,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:28.471480+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:24:28.471480+0000","last_peered":"2023-10-03T21:24:28.471480+0000","last_clean":"2023-10-03T21:24:28.471480+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:24:28.471480+0000","last_undegraded":"2023-10-03T21:24:28.471480+0000","last_fullsized":"2023-10-03T21:24:28.471480+0000","mapping_epoch":12,"log_start":"90'202313","ondisk_log_start":"90'202313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4951478606,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":213058,"num_read_kb":832706,"num_write":211243,"num_write_kb":10033282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'212520","reported_seq":428159,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:28.486585+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:24:28.486585+0000","last_peered":"2023-10-03T21:24:28.486585+0000","last_clean":"2023-10-03T21:24:28.486585+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:24:28.486585+0000","last_undegraded":"2023-10-03T21:24:28.486585+0000","last_fullsized":"2023-10-03T21:24:28.486585+0000","mapping_epoch":12,"log_start":"90'202513","ondisk_log_start":"90'202513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5009707327,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":215766,"num_read_kb":839672,"num_write":211344,"num_write_kb":9534056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'230930","reported_seq":463813,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:28.460446+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:24:28.460446+0000","last_peered":"2023-10-03T21:24:28.460446+0000","last_clean":"2023-10-03T21:24:28.460446+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:24:28.460446+0000","last_undegraded":"2023-10-03T21:24:28.460446+0000","last_fullsized":"2023-10-03T21:24:28.460446+0000","mapping_epoch":12,"log_start":"90'220908","ondisk_log_start":"90'220908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5222199799,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":232666,"num_read_kb":910449,"num_write":229684,"num_write_kb":9105517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'221250","reported_seq":444316,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:28.457040+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:24:28.457040+0000","last_peered":"2023-10-03T21:24:28.457040+0000","last_clean":"2023-10-03T21:24:28.457040+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:24:28.457040+0000","last_undegraded":"2023-10-03T21:24:28.457040+0000","last_fullsized":"2023-10-03T21:24:28.457040+0000","mapping_epoch":12,"log_start":"90'211214","ondisk_log_start":"90'211214","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5003829731,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":222607,"num_read_kb":868466,"num_write":218720,"num_write_kb":8363477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'33894","reported_seq":55853,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:26.467086+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:24:26.467086+0000","last_peered":"2023-10-03T21:24:26.467086+0000","last_clean":"2023-10-03T21:24:26.467086+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:24:26.467086+0000","last_undegraded":"2023-10-03T21:24:26.467086+0000","last_fullsized":"2023-10-03T21:24:26.467086+0000","mapping_epoch":15,"log_start":"84'23827","ondisk_log_start":"84'23827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154852766,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18726,"num_read_kb":54607,"num_write":15662,"num_write_kb":52102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15708,"num_flush":4592,"num_flush_kb":18524504,"num_evict":2501,"num_evict_kb":10105053,"num_promote":2508,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'213054","reported_seq":426888,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:26.498553+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:24:26.498553+0000","last_peered":"2023-10-03T21:24:26.498553+0000","last_clean":"2023-10-03T21:24:26.498553+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:24:26.498553+0000","last_undegraded":"2023-10-03T21:24:26.498553+0000","last_fullsized":"2023-10-03T21:24:26.498553+0000","mapping_epoch":12,"log_start":"90'203011","ondisk_log_start":"90'203011","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4731769153,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":213417,"num_read_kb":833009,"num_write":210685,"num_write_kb":7245764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'33832","reported_seq":55064,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:28.257901+0000","last_change":"2023-10-03T20:19:04.308101+0000","last_active":"2023-10-03T21:24:28.257901+0000","last_peered":"2023-10-03T21:24:28.257901+0000","last_clean":"2023-10-03T21:24:28.257901+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:24:28.257901+0000","last_undegraded":"2023-10-03T21:24:28.257901+0000","last_fullsized":"2023-10-03T21:24:28.257901+0000","mapping_epoch":15,"log_start":"84'23831","ondisk_log_start":"84'23831","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146374192,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19561,"num_read_kb":52130,"num_write":14647,"num_write_kb":47501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15719,"num_flush":3455,"num_flush_kb":13876396,"num_evict":2617,"num_evict_kb":10524752,"num_promote":2628,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'223771","reported_seq":448648,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:27.139372+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:24:27.139372+0000","last_peered":"2023-10-03T21:24:27.139372+0000","last_clean":"2023-10-03T21:24:27.139372+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:24:27.139372+0000","last_undegraded":"2023-10-03T21:24:27.139372+0000","last_fullsized":"2023-10-03T21:24:27.139372+0000","mapping_epoch":12,"log_start":"90'213715","ondisk_log_start":"90'213715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5112324517,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":224466,"num_read_kb":877710,"num_write":221205,"num_write_kb":8131947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'29786","reported_seq":49545,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:26.478165+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:24:26.478165+0000","last_peered":"2023-10-03T21:24:26.478165+0000","last_clean":"2023-10-03T21:24:26.478165+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:24:26.478165+0000","last_undegraded":"2023-10-03T21:24:26.478165+0000","last_fullsized":"2023-10-03T21:24:26.478165+0000","mapping_epoch":15,"log_start":"84'19734","ondisk_log_start":"84'19734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154857808,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16684,"num_read_kb":52212,"num_write":14987,"num_write_kb":49411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16469,"num_flush":3839,"num_flush_kb":15487184,"num_evict":2550,"num_evict_kb":10320777,"num_promote":2561,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'214766","reported_seq":432359,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:27.114929+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:24:27.114929+0000","last_peered":"2023-10-03T21:24:27.114929+0000","last_clean":"2023-10-03T21:24:27.114929+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:24:27.114929+0000","last_undegraded":"2023-10-03T21:24:27.114929+0000","last_fullsized":"2023-10-03T21:24:27.114929+0000","mapping_epoch":12,"log_start":"90'204713","ondisk_log_start":"90'204713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":200,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 29s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4810473923,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":217197,"num_read_kb":848730,"num_write":213461,"num_write_kb":7198093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'41973","reported_seq":67964,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:28.374420+0000","last_change":"2023-10-03T20:19:04.218798+0000","last_active":"2023-10-03T21:24:28.374420+0000","last_peered":"2023-10-03T21:24:28.374420+0000","last_clean":"2023-10-03T21:24:28.374420+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:24:28.374420+0000","last_undegraded":"2023-10-03T21:24:28.374420+0000","last_fullsized":"2023-10-03T21:24:28.374420+0000","mapping_epoch":15,"log_start":"84'31931","ondisk_log_start":"84'31931","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150777062,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":24673,"num_read_kb":57741,"num_write":14921,"num_write_kb":48830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15388,"num_flush":3779,"num_flush_kb":15156937,"num_evict":2506,"num_evict_kb":10074699,"num_promote":2515,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'220830","reported_seq":441657,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:28.453680+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:24:28.453680+0000","last_peered":"2023-10-03T21:24:28.453680+0000","last_clean":"2023-10-03T21:24:28.453680+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:24:28.453680+0000","last_undegraded":"2023-10-03T21:24:28.453680+0000","last_fullsized":"2023-10-03T21:24:28.453680+0000","mapping_epoch":12,"log_start":"90'210807","ondisk_log_start":"90'210807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":85,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4938178944,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":220537,"num_read_kb":862542,"num_write":219585,"num_write_kb":10167267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":254,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T21:23:08.205421+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:23:08.205421+0000","last_peered":"2023-10-03T21:23:08.205421+0000","last_clean":"2023-10-03T21:23:08.205421+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:23:08.205421+0000","last_undegraded":"2023-10-03T21:23:08.205421+0000","last_fullsized":"2023-10-03T21:23:08.205421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606861828,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":79644,"num_read_kb":216690,"num_write":60217,"num_write_kb":197844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63284,"num_flush":15665,"num_flush_kb":63045021,"num_evict":10174,"num_evict_kb":41025281,"num_promote":10212,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4857,"num_omap_keys":76,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1209778176,"data_stored":1209488392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40162,"ondisk_log_size":40162,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39779962000,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1759714,"num_read_kb":6873284,"num_write":1735927,"num_write_kb":69779403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42233249792,"data_stored":47995124000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80329,"ondisk_log_size":80329,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739221,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9411140,"kb_used_data":8270844,"kb_used_omap":376,"kb_used_meta":1139911,"kb_avail":84960700,"statfs":{"total":96636764160,"available":86999756800,"internally_reserved":0,"allocated":8469344256,"data_stored":9661111960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1167269789},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":0.82899999999999996,"15min":1.026},"min":{"1min":0.193,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.0410000000000004,"5min":5.5670000000000002,"15min":19.114000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.81899999999999995,"15min":0.98399999999999999},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":5.9260000000000002,"5min":6.1660000000000004,"15min":15.244},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.94799999999999995,"15min":0.93899999999999995},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.2859999999999996,"5min":5.9749999999999996,"15min":30.677},"last":0.376},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.4630000000000001,"15min":1.3600000000000001},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.073,"5min":18.285,"15min":30.045999999999999},"last":0.31900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739212,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22543812,"kb_used_data":20905144,"kb_used_omap":961,"kb_used_meta":1637694,"kb_avail":71828028,"statfs":{"total":96636764160,"available":73551900672,"internally_reserved":0,"allocated":21406867456,"data_stored":24287291762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1676998693},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,9,15,10,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.4770000000000001,"15min":2.2170000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.13800000000000001},"max":{"1min":6.5599999999999996,"5min":15.569000000000001,"15min":28.088999999999999},"last":1.643},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.232,"15min":1.3129999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":4.7430000000000003,"5min":28.699999999999999,"15min":28.699999999999999},"last":3.8029999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:23:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.369,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":5.0090000000000003,"5min":28.585999999999999,"15min":28.585999999999999},"last":8.3230000000000004},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.278,"15min":1.202},"min":{"1min":0.21099999999999999,"5min":0.20999999999999999,"15min":0.20699999999999999},"max":{"1min":6.2370000000000001,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.23100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14939924,"kb_used_data":13239052,"kb_used_omap":597,"kb_used_meta":1700266,"kb_avail":79431916,"statfs":{"total":96636764160,"available":81338281984,"internally_reserved":0,"allocated":13556789248,"data_stored":15245105250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1741073313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,3,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:23:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.89600000000000002,"15min":1.2889999999999999},"min":{"1min":0.27100000000000002,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":3.407,"5min":6.601,"15min":21.391999999999999},"last":0.437},{"interface":"front","average":{"1min":1.083,"5min":0.95299999999999996,"15min":1.046},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":6.2990000000000004,"5min":11.045999999999999,"15min":21.204999999999998},"last":0.25800000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:23:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0489999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":6.2320000000000002,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.91300000000000003,"15min":0.85199999999999998},"min":{"1min":0.25900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.5459999999999998,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13245931520,"data_stored":14934673320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21117124608,"data_stored":23997975696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7870193664,"data_stored":9062474984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309768192,"data_stored":309698286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297222144,"data_stored":297147158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602787840,"data_stored":602642948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:24:31.630 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.630 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.630 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.631 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.631 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.631 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.631 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.631 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:31.631 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:24:34.632 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.625+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:24:34.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.633+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:24:34.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.633+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:24:34.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.633+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:24:34.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.633+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:24:34.650 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.641+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:24:34.650 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:34.641+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:24:40.444 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:40.437+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:24:40.488 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:40.481+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:24:40.492 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:40.481+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:24:40.492 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:40.481+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:24:51.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:24:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:24:52.245 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:24:52.300 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2237,"stamp":"2023-10-03T21:24:52.188487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40439441492,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":1847350,"num_read_kb":7121289,"num_write":1804002,"num_write_kb":70311823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63239,"num_flush":15743,"num_flush_kb":63363593,"num_evict":10227,"num_evict_kb":41237802,"num_promote":10264,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20377,"num_omap_keys":352,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120617,"ondisk_log_size":120617,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":47193316,"kb_used_data":42554248,"kb_used_omap":1935,"kb_used_meta":4637104,"kb_avail":235922204,"statfs":{"total":289910292480,"available":241584336896,"internally_reserved":0,"allocated":43575549952,"data_stored":49317207959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4748395363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,5,4,19,25,25,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14430178,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":4117,"num_read_kb":16296,"num_write":4041,"num_write_kb":187494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":21,"num_flush":46,"num_flush_kb":187948,"num_evict":33,"num_evict_kb":130781,"num_promote":28,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":-1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002178"},"pg_stats":[{"pgid":"2.7","version":"93'213229","reported_seq":427444,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:48.840500+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:24:48.840500+0000","last_peered":"2023-10-03T21:24:48.840500+0000","last_clean":"2023-10-03T21:24:48.840500+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:24:48.840500+0000","last_undegraded":"2023-10-03T21:24:48.840500+0000","last_fullsized":"2023-10-03T21:24:48.840500+0000","mapping_epoch":12,"log_start":"90'203213","ondisk_log_start":"90'203213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4967858510,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":213968,"num_read_kb":836318,"num_write":212070,"num_write_kb":10097774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'213384","reported_seq":429967,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:48.801847+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:24:48.801847+0000","last_peered":"2023-10-03T21:24:48.801847+0000","last_clean":"2023-10-03T21:24:48.801847+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:24:48.801847+0000","last_undegraded":"2023-10-03T21:24:48.801847+0000","last_fullsized":"2023-10-03T21:24:48.801847+0000","mapping_epoch":12,"log_start":"90'203313","ondisk_log_start":"90'203313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5017239871,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":216700,"num_read_kb":843348,"num_write":212208,"num_write_kb":9578356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'231947","reported_seq":465816,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:48.814499+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:24:48.814499+0000","last_peered":"2023-10-03T21:24:48.814499+0000","last_clean":"2023-10-03T21:24:48.814499+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:24:48.814499+0000","last_undegraded":"2023-10-03T21:24:48.814499+0000","last_fullsized":"2023-10-03T21:24:48.814499+0000","mapping_epoch":12,"log_start":"90'221908","ondisk_log_start":"90'221908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5229904375,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":233647,"num_read_kb":914341,"num_write":230701,"num_write_kb":9158541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'222152","reported_seq":446115,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:48.824989+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:24:48.824989+0000","last_peered":"2023-10-03T21:24:48.824989+0000","last_clean":"2023-10-03T21:24:48.824989+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:24:48.824989+0000","last_undegraded":"2023-10-03T21:24:48.824989+0000","last_fullsized":"2023-10-03T21:24:48.824989+0000","mapping_epoch":12,"log_start":"90'212114","ondisk_log_start":"90'212114","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5008458211,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":223501,"num_read_kb":872014,"num_write":219622,"num_write_kb":8383401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'34081","reported_seq":56156,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:51.382441+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:24:51.382441+0000","last_peered":"2023-10-03T21:24:51.382441+0000","last_clean":"2023-10-03T21:24:51.382441+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:24:51.382441+0000","last_undegraded":"2023-10-03T21:24:51.382441+0000","last_fullsized":"2023-10-03T21:24:51.382441+0000","mapping_epoch":15,"log_start":"84'24028","ondisk_log_start":"84'24028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159083938,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":18799,"num_read_kb":54899,"num_write":15761,"num_write_kb":52430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15712,"num_flush":4617,"num_flush_kb":18626560,"num_evict":2517,"num_evict_kb":10170421,"num_promote":2525,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'214112","reported_seq":429058,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:51.546180+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:24:51.546180+0000","last_peered":"2023-10-03T21:24:51.546180+0000","last_clean":"2023-10-03T21:24:51.546180+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:24:51.546180+0000","last_undegraded":"2023-10-03T21:24:51.546180+0000","last_fullsized":"2023-10-03T21:24:51.546180+0000","mapping_epoch":12,"log_start":"90'204111","ondisk_log_start":"90'204111","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4736782657,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":214521,"num_read_kb":837357,"num_write":211743,"num_write_kb":7290848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'33948","reported_seq":55284,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:48.387115+0000","last_change":"2023-10-03T21:24:38.578536+0000","last_active":"2023-10-03T21:24:48.387115+0000","last_peered":"2023-10-03T21:24:48.387115+0000","last_clean":"2023-10-03T21:24:48.387115+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:24:48.387115+0000","last_undegraded":"2023-10-03T21:24:48.387115+0000","last_fullsized":"2023-10-03T21:24:48.387115+0000","mapping_epoch":15,"log_start":"84'23931","ondisk_log_start":"84'23931","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":23,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 9s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154738207,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19622,"num_read_kb":52374,"num_write":14702,"num_write_kb":47695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15702,"num_flush":3468,"num_flush_kb":13929452,"num_evict":2623,"num_evict_kb":10549248,"num_promote":2636,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'224691","reported_seq":450511,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:47.391801+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:24:47.391801+0000","last_peered":"2023-10-03T21:24:47.391801+0000","last_clean":"2023-10-03T21:24:47.391801+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:24:47.391801+0000","last_undegraded":"2023-10-03T21:24:47.391801+0000","last_fullsized":"2023-10-03T21:24:47.391801+0000","mapping_epoch":12,"log_start":"90'214615","ondisk_log_start":"90'214615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5116842405,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":225404,"num_read_kb":881430,"num_write":222125,"num_write_kb":8160143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'29922","reported_seq":49783,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:51.469047+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:24:51.469047+0000","last_peered":"2023-10-03T21:24:51.469047+0000","last_clean":"2023-10-03T21:24:51.469047+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:24:51.469047+0000","last_undegraded":"2023-10-03T21:24:51.469047+0000","last_fullsized":"2023-10-03T21:24:51.469047+0000","mapping_epoch":15,"log_start":"84'19834","ondisk_log_start":"84'19834","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":142356778,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":3,"num_read":16750,"num_read_kb":52472,"num_write":15057,"num_write_kb":49629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16431,"num_flush":3860,"num_flush_kb":15572960,"num_evict":2565,"num_evict_kb":10382093,"num_promote":2573,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'215729","reported_seq":434263,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:47.392047+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:24:47.392047+0000","last_peered":"2023-10-03T21:24:47.392047+0000","last_clean":"2023-10-03T21:24:47.392047+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:24:47.392047+0000","last_undegraded":"2023-10-03T21:24:47.392047+0000","last_fullsized":"2023-10-03T21:24:47.392047+0000","mapping_epoch":12,"log_start":"90'205713","ondisk_log_start":"90'205713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":325,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 49s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4814987715,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":218122,"num_read_kb":852410,"num_write":214340,"num_write_kb":7238297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'42158","reported_seq":68288,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:48.728854+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:24:48.728854+0000","last_peered":"2023-10-03T21:24:48.728854+0000","last_clean":"2023-10-03T21:24:48.728854+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:24:48.728854+0000","last_undegraded":"2023-10-03T21:24:48.728854+0000","last_fullsized":"2023-10-03T21:24:48.728854+0000","mapping_epoch":15,"log_start":"84'32131","ondisk_log_start":"84'32131","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150768825,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":24742,"num_read_kb":58013,"num_write":14994,"num_write_kb":49056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15394,"num_flush":3798,"num_flush_kb":15234621,"num_evict":2522,"num_evict_kb":10136040,"num_promote":2530,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'221885","reported_seq":443666,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:24:48.828561+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:24:48.828561+0000","last_peered":"2023-10-03T21:24:48.828561+0000","last_clean":"2023-10-03T21:24:48.828561+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:24:48.828561+0000","last_undegraded":"2023-10-03T21:24:48.828561+0000","last_fullsized":"2023-10-03T21:24:48.828561+0000","mapping_epoch":12,"log_start":"90'211807","ondisk_log_start":"90'211807","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":226,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 29s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4939829632,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":221468,"num_read_kb":866222,"num_write":220542,"num_write_kb":10203759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":255,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:23:56.814386+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:23:56.814386+0000","last_peered":"2023-10-03T21:23:56.814386+0000","last_clean":"2023-10-03T21:23:56.814386+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:23:56.814386+0000","last_undegraded":"2023-10-03T21:23:56.814386+0000","last_fullsized":"2023-10-03T21:23:56.814386+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606947748,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":79913,"num_read_kb":217758,"num_write":60514,"num_write_kb":198810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63239,"num_flush":15743,"num_flush_kb":63363593,"num_evict":10227,"num_evict_kb":41237802,"num_promote":10264,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10223,"num_omap_keys":162,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218269184,"data_stored":1217987379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40185,"ondisk_log_size":40185,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39831903376,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1767331,"num_read_kb":6903440,"num_write":1743351,"num_write_kb":70111119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42341916672,"data_stored":48084994336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80335,"ondisk_log_size":80335,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9464420,"kb_used_data":8293596,"kb_used_omap":376,"kb_used_meta":1170439,"kb_avail":84907420,"statfs":{"total":96636764160,"available":86945198080,"internally_reserved":0,"allocated":8492642304,"data_stored":9682791971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1198530461},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":0.82899999999999996,"15min":1.026},"min":{"1min":0.193,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.0410000000000004,"5min":5.5670000000000002,"15min":19.114000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.81899999999999995,"15min":0.98399999999999999},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":5.9260000000000002,"5min":6.1660000000000004,"15min":15.244},"last":0.246}]},{"osd":1,"last update":"Tue Oct 3 21:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.94799999999999995,"15min":0.93899999999999995},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.2859999999999996,"5min":5.9749999999999996,"15min":30.677},"last":0.77200000000000002},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.4630000000000001,"15min":1.3600000000000001},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.073,"5min":18.285,"15min":30.045999999999999},"last":0.22600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22693132,"kb_used_data":20972612,"kb_used_omap":961,"kb_used_meta":1719550,"kb_avail":71678708,"statfs":{"total":96636764160,"available":73398996992,"internally_reserved":0,"allocated":21475954688,"data_stored":24347220276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1760819237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,1,13,11,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:24:34 2023","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":1.266,"15min":2.21},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.153},"max":{"1min":11.055999999999999,"5min":11.055999999999999,"15min":28.088999999999999},"last":5.423},{"interface":"front","average":{"1min":1.075,"5min":1.3180000000000001,"15min":1.3089999999999999},"min":{"1min":0.26000000000000001,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":3.8029999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":6.2709999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:24:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.472,"15min":1.3080000000000001},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":8.3230000000000004,"5min":28.585999999999999,"15min":28.585999999999999},"last":6.3079999999999998},{"interface":"front","average":{"1min":0.879,"5min":1.3740000000000001,"15min":1.212},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":3.7629999999999999,"5min":14.201000000000001,"15min":14.201000000000001},"last":4.9059999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15035764,"kb_used_data":13288040,"kb_used_omap":597,"kb_used_meta":1747114,"kb_avail":79336076,"statfs":{"total":96636764160,"available":81240141824,"internally_reserved":0,"allocated":13606952960,"data_stored":15287195712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1789045665},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,3,7,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:24:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":1.224},"min":{"1min":0.182,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":5.9390000000000001,"5min":5.9390000000000001,"15min":21.391999999999999},"last":2.7919999999999998},{"interface":"front","average":{"1min":1.982,"5min":1.0740000000000001,"15min":1.123},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":23.422000000000001,"5min":23.422000000000001,"15min":23.422000000000001},"last":3.0259999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:24:33 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.052,"15min":1.119},"min":{"1min":0.222,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":6.0940000000000003,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97399999999999998,"15min":0.88400000000000001},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.189},"max":{"1min":5.9900000000000002,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13291859968,"data_stored":14972549032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21169143808,"data_stored":24040873104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7880912896,"data_stored":9071572200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301498368,"data_stored":301428428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305573888,"data_stored":305502936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":611196928,"data_stored":611056015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:24:52.302 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.302 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.302 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.302 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.303 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.303 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.303 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:24:52.303 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:24:52.898 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:52.892+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:24:52.907 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:52.896+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:24:52.915 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:52.908+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:24:52.916 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:24:52.908+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:24:55.170 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:55.160+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:24:55.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:55.160+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:24:55.171 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:55.164+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:24:58.840 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:58.836+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:24:58.841 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:58.836+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:24:58.870 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:58.864+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:24:58.871 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:24:58.864+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:25:03.408 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:03.404+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:25:03.417 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:03.412+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:25:04.179 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.172+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:25:04.184 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.172+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:25:04.186 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.180+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:25:04.186 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.180+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:25:04.203 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.184+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:25:04.203 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.188+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:25:04.203 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:04.188+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:25:12.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:25:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:25:12.913 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:25:12.971 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2248,"stamp":"2023-10-03T21:25:12.193021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40493897824,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1854837,"num_read_kb":7150877,"num_write":1811480,"num_write_kb":70619080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63251,"num_flush":15822,"num_flush_kb":63686421,"num_evict":10276,"num_evict_kb":41438146,"num_promote":10313,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20377,"num_omap_keys":352,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120880,"ondisk_log_size":120880,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":47457596,"kb_used_data":42639064,"kb_used_omap":1935,"kb_used_meta":4816560,"kb_avail":235657924,"statfs":{"total":289910292480,"available":241313714176,"internally_reserved":0,"allocated":43662401536,"data_stored":49367743794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4932158307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,7,17,13,37,17,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13602781,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3816,"num_read_kb":15096,"num_write":3733,"num_write_kb":125034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-35,"num_flush":38,"num_flush_kb":155220,"num_evict":27,"num_evict_kb":110428,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001970"},"pg_stats":[{"pgid":"2.7","version":"94'214061","reported_seq":429145,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:09.070417+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:25:09.070417+0000","last_peered":"2023-10-03T21:25:09.070417+0000","last_clean":"2023-10-03T21:25:09.070417+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:25:09.070417+0000","last_undegraded":"2023-10-03T21:25:09.070417+0000","last_fullsized":"2023-10-03T21:25:09.070417+0000","mapping_epoch":12,"log_start":"90'204013","ondisk_log_start":"90'204013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4984230222,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":214831,"num_read_kb":839722,"num_write":212902,"num_write_kb":10182706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'214302","reported_seq":431805,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:09.085082+0000","last_change":"2023-10-03T21:20:15.180473+0000","last_active":"2023-10-03T21:25:09.085082+0000","last_peered":"2023-10-03T21:25:09.085082+0000","last_clean":"2023-10-03T21:25:09.085082+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:25:09.085082+0000","last_undegraded":"2023-10-03T21:25:09.085082+0000","last_fullsized":"2023-10-03T21:25:09.085082+0000","mapping_epoch":12,"log_start":"90'204213","ondisk_log_start":"90'204213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5024948543,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":217612,"num_read_kb":846952,"num_write":213126,"num_write_kb":9626948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'232862","reported_seq":467641,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:09.102043+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:25:09.102043+0000","last_peered":"2023-10-03T21:25:09.102043+0000","last_clean":"2023-10-03T21:25:09.102043+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:25:09.102043+0000","last_undegraded":"2023-10-03T21:25:09.102043+0000","last_fullsized":"2023-10-03T21:25:09.102043+0000","mapping_epoch":12,"log_start":"91'222808","ondisk_log_start":"91'222808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5231391223,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":234548,"num_read_kb":917909,"num_write":231616,"num_write_kb":9174461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'223072","reported_seq":448013,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:09.100978+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:25:09.100978+0000","last_peered":"2023-10-03T21:25:09.100978+0000","last_clean":"2023-10-03T21:25:09.100978+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:25:09.100978+0000","last_undegraded":"2023-10-03T21:25:09.100978+0000","last_fullsized":"2023-10-03T21:25:09.100978+0000","mapping_epoch":12,"log_start":"91'213014","ondisk_log_start":"91'213014","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5010010595,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":224472,"num_read_kb":875842,"num_write":220542,"num_write_kb":8399317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'34261","reported_seq":56435,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:11.779193+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:25:11.779193+0000","last_peered":"2023-10-03T21:25:11.779193+0000","last_clean":"2023-10-03T21:25:11.779193+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:25:11.779193+0000","last_undegraded":"2023-10-03T21:25:11.779193+0000","last_fullsized":"2023-10-03T21:25:11.779193+0000","mapping_epoch":15,"log_start":"84'24228","ondisk_log_start":"84'24228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154799487,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":18849,"num_read_kb":55099,"num_write":15839,"num_write_kb":52700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15677,"num_flush":4649,"num_flush_kb":18757356,"num_evict":2527,"num_evict_kb":10211353,"num_promote":2534,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'215010","reported_seq":430876,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:11.788677+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:25:11.788677+0000","last_peered":"2023-10-03T21:25:11.788677+0000","last_clean":"2023-10-03T21:25:11.788677+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:25:11.788677+0000","last_undegraded":"2023-10-03T21:25:11.788677+0000","last_fullsized":"2023-10-03T21:25:11.788677+0000","mapping_epoch":12,"log_start":"91'204911","ondisk_log_start":"91'204911","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4741382465,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":215435,"num_read_kb":840973,"num_write":212641,"num_write_kb":7314884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'34121","reported_seq":55550,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:09.089728+0000","last_change":"2023-10-03T21:24:38.578536+0000","last_active":"2023-10-03T21:25:09.089728+0000","last_peered":"2023-10-03T21:25:09.089728+0000","last_clean":"2023-10-03T21:25:09.089728+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:25:09.089728+0000","last_undegraded":"2023-10-03T21:25:09.089728+0000","last_fullsized":"2023-10-03T21:25:09.089728+0000","mapping_epoch":15,"log_start":"84'24031","ondisk_log_start":"84'24031","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:31.145107+0000","last_clean_scrub_stamp":"2023-10-03T20:13:31.145107+0000","objects_scrubbed":53,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 30s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154725932,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19674,"num_read_kb":52582,"num_write":14761,"num_write_kb":47881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15715,"num_flush":3483,"num_flush_kb":13990644,"num_evict":2635,"num_evict_kb":10598276,"num_promote":2648,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"94'225589","reported_seq":452341,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:07.472071+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:25:07.472071+0000","last_peered":"2023-10-03T21:25:07.472071+0000","last_clean":"2023-10-03T21:25:07.472071+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:25:07.472071+0000","last_undegraded":"2023-10-03T21:25:07.472071+0000","last_fullsized":"2023-10-03T21:25:07.472071+0000","mapping_epoch":12,"log_start":"91'215515","ondisk_log_start":"91'215515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5121368485,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":226326,"num_read_kb":885070,"num_write":223023,"num_write_kb":8200451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'30068","reported_seq":50010,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:11.640611+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:25:11.640611+0000","last_peered":"2023-10-03T21:25:11.640611+0000","last_clean":"2023-10-03T21:25:11.640611+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:25:11.640611+0000","last_undegraded":"2023-10-03T21:25:11.640611+0000","last_fullsized":"2023-10-03T21:25:11.640611+0000","mapping_epoch":15,"log_start":"84'20034","ondisk_log_start":"84'20034","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146501976,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16803,"num_read_kb":52684,"num_write":15123,"num_write_kb":49836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16477,"num_flush":3876,"num_flush_kb":15638372,"num_evict":2579,"num_evict_kb":10439357,"num_promote":2588,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"94'216680","reported_seq":436149,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:07.501451+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:25:07.501451+0000","last_peered":"2023-10-03T21:25:07.501451+0000","last_clean":"2023-10-03T21:25:07.501451+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:25:07.501451+0000","last_undegraded":"2023-10-03T21:25:07.501451+0000","last_fullsized":"2023-10-03T21:25:07.501451+0000","mapping_epoch":12,"log_start":"90'206613","ondisk_log_start":"90'206613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":457,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 69s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4822770115,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":219038,"num_read_kb":856038,"num_write":215203,"num_write_kb":7282525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'42374","reported_seq":68603,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:08.692617+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:25:08.692617+0000","last_peered":"2023-10-03T21:25:08.692617+0000","last_clean":"2023-10-03T21:25:08.692617+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:25:08.692617+0000","last_undegraded":"2023-10-03T21:25:08.692617+0000","last_fullsized":"2023-10-03T21:25:08.692617+0000","mapping_epoch":15,"log_start":"84'32331","ondisk_log_start":"84'32331","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150781101,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24800,"num_read_kb":58245,"num_write":15062,"num_write_kb":49274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15382,"num_flush":3814,"num_flush_kb":15300049,"num_evict":2535,"num_evict_kb":10189160,"num_promote":2543,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'223001","reported_seq":445680,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:09.102958+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:25:09.102958+0000","last_peered":"2023-10-03T21:25:09.102958+0000","last_clean":"2023-10-03T21:25:09.102958+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:25:09.102958+0000","last_undegraded":"2023-10-03T21:25:09.102958+0000","last_fullsized":"2023-10-03T21:25:09.102958+0000","mapping_epoch":12,"log_start":"91'212907","ondisk_log_start":"91'212907","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":457,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 49s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4950397312,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":222343,"num_read_kb":869670,"num_write":221505,"num_write_kb":10236203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":256,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T21:24:38.652790+0000","last_change":"2023-10-03T21:20:16.661774+0000","last_active":"2023-10-03T21:24:38.652790+0000","last_peered":"2023-10-03T21:24:38.652790+0000","last_clean":"2023-10-03T21:24:38.652790+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:24:38.652790+0000","last_undegraded":"2023-10-03T21:24:38.652790+0000","last_fullsized":"2023-10-03T21:24:38.652790+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:21.072362+0000","last_clean_scrub_stamp":"2023-10-03T20:13:21.072362+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606808496,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":80126,"num_read_kb":218610,"num_write":60785,"num_write_kb":199691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63251,"num_flush":15822,"num_flush_kb":63686421,"num_evict":10276,"num_evict_kb":41438146,"num_promote":10313,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10223,"num_omap_keys":162,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1209774080,"data_stored":1209492355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39886498960,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1774605,"num_read_kb":6932176,"num_write":1750558,"num_write_kb":70417495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42461855744,"data_stored":48168687904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80583,"ondisk_log_size":80583,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739229,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9497740,"kb_used_data":8304260,"kb_used_omap":376,"kb_used_meta":1193095,"kb_avail":84874100,"statfs":{"total":96636764160,"available":86911078400,"internally_reserved":0,"allocated":8503562240,"data_stored":9685392768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1221730205},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":0.82899999999999996,"15min":1.026},"min":{"1min":0.193,"5min":0.16900000000000001,"15min":0.14899999999999999},"max":{"1min":5.0410000000000004,"5min":5.5670000000000002,"15min":19.114000000000001},"last":3.1960000000000002},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.81899999999999995,"15min":0.98399999999999999},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":5.9260000000000002,"5min":6.1660000000000004,"15min":15.244},"last":3.0569999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:24:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.94799999999999995,"15min":0.93899999999999995},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":5.2859999999999996,"5min":5.9749999999999996,"15min":30.677},"last":3.653},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.4630000000000001,"15min":1.3600000000000001},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.073,"5min":18.285,"15min":30.045999999999999},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739220,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22837360,"kb_used_data":21018332,"kb_used_omap":961,"kb_used_meta":1818046,"kb_avail":71534480,"statfs":{"total":96636764160,"available":73251307520,"internally_reserved":0,"allocated":21522771968,"data_stored":24379181134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1861679141},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,3,4,21,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:24:34 2023","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":1.266,"15min":2.21},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.153},"max":{"1min":11.055999999999999,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.075,"5min":1.3180000000000001,"15min":1.3089999999999999},"min":{"1min":0.26000000000000001,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":3.8029999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":1.069}]},{"osd":2,"last update":"Tue Oct 3 21:24:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.472,"15min":1.3080000000000001},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":8.3230000000000004,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.3740000000000001,"15min":1.212},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":3.7629999999999999,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.34300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15122496,"kb_used_data":13316472,"kb_used_omap":597,"kb_used_meta":1805418,"kb_avail":79249344,"statfs":{"total":96636764160,"available":81151328256,"internally_reserved":0,"allocated":13636067328,"data_stored":15303169892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1848748961},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,9,8,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:24:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":1.224},"min":{"1min":0.182,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":5.9390000000000001,"5min":5.9390000000000001,"15min":21.391999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.982,"5min":1.0740000000000001,"15min":1.123},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":23.422000000000001,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:24:33 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.052,"15min":1.119},"min":{"1min":0.222,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":6.0940000000000003,"5min":8.8559999999999999,"15min":10.914999999999999},"last":0.223},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97399999999999998,"15min":0.88400000000000001},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.189},"max":{"1min":5.9900000000000002,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13333344256,"data_stored":15000934312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21224366080,"data_stored":24081222800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7904145408,"data_stored":9086530792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301359104,"data_stored":301289175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305573888,"data_stored":305502937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602841088,"data_stored":602700243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:25:12.973 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.973 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.973 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.974 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T20:13:31.145107+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=31, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.974 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.974 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.974 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T20:13:21.072362+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=21, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:12.974 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:25:12.991 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:12.984+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:25:13.114 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:13.108+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:25:13.120 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:13.112+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:25:13.154 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:13.148+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:25:13.161 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:13.156+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:25:13.161 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:13.156+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:25:13.161 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:13.156+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:25:15.258 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:15.252+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:25:15.266 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:15.260+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:25:15.266 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:15.260+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:25:15.272 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:15.264+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:25:15.287 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:15.280+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:25:15.287 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:15.280+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:25:15.482 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.476+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:25:15.482 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.476+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:25:15.483 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.476+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:25:15.490 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.484+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:25:15.496 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.488+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:25:15.496 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.488+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:25:15.496 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:15.488+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:25:21.285 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:21.280+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:25:21.290 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:21.284+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:25:28.237 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:28.228+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:25:28.256 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:28.248+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:25:28.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:28.248+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:25:28.257 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:28.248+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:25:32.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:25:33.930 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:25:33.932 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:25:33.986 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2259,"stamp":"2023-10-03T21:25:32.197543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40559343772,"num_objects":14920,"num_object_clones":9472,"num_object_copies":29840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14796,"num_whiteouts":11,"num_read":1862372,"num_read_kb":7180677,"num_write":1818984,"num_write_kb":71000048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63311,"num_flush":15894,"num_flush_kb":63980773,"num_evict":10330,"num_evict_kb":41658830,"num_promote":10366,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120923,"ondisk_log_size":120923,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":47635652,"kb_used_data":42779620,"kb_used_omap":1935,"kb_used_meta":4854064,"kb_avail":235479868,"statfs":{"total":289910292480,"available":241131384832,"internally_reserved":0,"allocated":43806330880,"data_stored":49480788534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":4970562403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,11,17,21,23,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24260635,"num_objects":-3,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3940,"num_read_kb":15584,"num_write":3995,"num_write_kb":191462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":27,"num_flush":41,"num_flush_kb":167624,"num_evict":37,"num_evict_kb":151156,"num_promote":34,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.398705"},"pg_stats":[{"pgid":"2.7","version":"95'214903","reported_seq":430913,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:29.569794+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:25:29.569794+0000","last_peered":"2023-10-03T21:25:29.569794+0000","last_clean":"2023-10-03T21:25:29.569794+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:25:29.569794+0000","last_undegraded":"2023-10-03T21:25:29.569794+0000","last_fullsized":"2023-10-03T21:25:29.569794+0000","mapping_epoch":12,"log_start":"91'204813","ondisk_log_start":"91'204813","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4991549774,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":215748,"num_read_kb":843334,"num_write":213744,"num_write_kb":10247326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'215249","reported_seq":433677,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:29.636796+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:25:29.636796+0000","last_peered":"2023-10-03T21:25:29.636796+0000","last_clean":"2023-10-03T21:25:29.636796+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:25:29.636796+0000","last_undegraded":"2023-10-03T21:25:29.636796+0000","last_fullsized":"2023-10-03T21:25:29.636796+0000","mapping_epoch":12,"log_start":"91'205213","ondisk_log_start":"91'205213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":112,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5032239423,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":218515,"num_read_kb":850520,"num_write":213995,"num_write_kb":9675384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'233887","reported_seq":469612,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:29.630303+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:25:29.630303+0000","last_peered":"2023-10-03T21:25:29.630303+0000","last_clean":"2023-10-03T21:25:29.630303+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:25:29.630303+0000","last_undegraded":"2023-10-03T21:25:29.630303+0000","last_fullsized":"2023-10-03T21:25:29.630303+0000","mapping_epoch":12,"log_start":"91'223808","ondisk_log_start":"91'223808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5239161335,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":235487,"num_read_kb":921629,"num_write":232641,"num_write_kb":9227557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'223996","reported_seq":449871,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:29.623084+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:25:29.623084+0000","last_peered":"2023-10-03T21:25:29.623084+0000","last_clean":"2023-10-03T21:25:29.623084+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:25:29.623084+0000","last_undegraded":"2023-10-03T21:25:29.623084+0000","last_fullsized":"2023-10-03T21:25:29.623084+0000","mapping_epoch":12,"log_start":"91'213914","ondisk_log_start":"91'213914","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5029855715,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":225398,"num_read_kb":879486,"num_write":221466,"num_write_kb":8472429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'34418","reported_seq":56669,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:31.796311+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:25:31.796311+0000","last_peered":"2023-10-03T21:25:31.796311+0000","last_clean":"2023-10-03T21:25:31.796311+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:25:31.796311+0000","last_undegraded":"2023-10-03T21:25:31.796311+0000","last_fullsized":"2023-10-03T21:25:31.796311+0000","mapping_epoch":15,"log_start":"84'24328","ondisk_log_start":"84'24328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158981530,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":5,"num_read":18890,"num_read_kb":55259,"num_write":15917,"num_write_kb":52962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15704,"num_flush":4668,"num_flush_kb":18835024,"num_evict":2539,"num_evict_kb":10260417,"num_promote":2547,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'215887","reported_seq":432635,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:31.969478+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:25:31.969478+0000","last_peered":"2023-10-03T21:25:31.969478+0000","last_clean":"2023-10-03T21:25:31.969478+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:25:31.969478+0000","last_undegraded":"2023-10-03T21:25:31.969478+0000","last_fullsized":"2023-10-03T21:25:31.969478+0000","mapping_epoch":12,"log_start":"91'205811","ondisk_log_start":"91'205811","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4749107521,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":216310,"num_read_kb":844449,"num_write":213518,"num_write_kb":7338828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'34295","reported_seq":55818,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:29.178430+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:25:29.178430+0000","last_peered":"2023-10-03T21:25:29.178430+0000","last_clean":"2023-10-03T21:25:29.178430+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:25:29.178430+0000","last_undegraded":"2023-10-03T21:25:29.178430+0000","last_fullsized":"2023-10-03T21:25:29.178430+0000","mapping_epoch":15,"log_start":"84'24232","ondisk_log_start":"84'24232","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146419275,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19724,"num_read_kb":52782,"num_write":14838,"num_write_kb":48119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15746,"num_flush":3501,"num_flush_kb":14064220,"num_evict":2652,"num_evict_kb":10667720,"num_promote":2663,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'226545","reported_seq":454288,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:27.701246+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:25:27.701246+0000","last_peered":"2023-10-03T21:25:27.701246+0000","last_clean":"2023-10-03T21:25:27.701246+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:25:27.701246+0000","last_undegraded":"2023-10-03T21:25:27.701246+0000","last_fullsized":"2023-10-03T21:25:27.701246+0000","mapping_epoch":12,"log_start":"91'216515","ondisk_log_start":"91'216515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5132095909,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":227313,"num_read_kb":888978,"num_write":223979,"num_write_kb":8253263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'30198","reported_seq":50218,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:31.733721+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:25:31.733721+0000","last_peered":"2023-10-03T21:25:31.733721+0000","last_clean":"2023-10-03T21:25:31.733721+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:25:31.733721+0000","last_undegraded":"2023-10-03T21:25:31.733721+0000","last_fullsized":"2023-10-03T21:25:31.733721+0000","mapping_epoch":15,"log_start":"84'20134","ondisk_log_start":"84'20134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146383185,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16850,"num_read_kb":52872,"num_write":15189,"num_write_kb":50050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16470,"num_flush":3895,"num_flush_kb":15716020,"num_evict":2591,"num_evict_kb":10488369,"num_promote":2600,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'217614","reported_seq":437982,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:27.677010+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:25:27.677010+0000","last_peered":"2023-10-03T21:25:27.677010+0000","last_clean":"2023-10-03T21:25:27.677010+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:25:27.677010+0000","last_undegraded":"2023-10-03T21:25:27.677010+0000","last_fullsized":"2023-10-03T21:25:27.677010+0000","mapping_epoch":12,"log_start":"91'207514","ondisk_log_start":"91'207514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":608,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 89s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4824162755,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":219919,"num_read_kb":859530,"num_write":216034,"num_write_kb":7298057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'42558","reported_seq":68879,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:29.418983+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:25:29.418983+0000","last_peered":"2023-10-03T21:25:29.418983+0000","last_clean":"2023-10-03T21:25:29.418983+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:25:29.418983+0000","last_undegraded":"2023-10-03T21:25:29.418983+0000","last_fullsized":"2023-10-03T21:25:29.418983+0000","mapping_epoch":15,"log_start":"84'32531","ondisk_log_start":"84'32531","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150727862,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":24844,"num_read_kb":58421,"num_write":15122,"num_write_kb":49460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15391,"num_flush":3830,"num_flush_kb":15365509,"num_evict":2548,"num_evict_kb":10242324,"num_promote":2556,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'223995","reported_seq":447622,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:29.610977+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:25:29.610977+0000","last_peered":"2023-10-03T21:25:29.610977+0000","last_clean":"2023-10-03T21:25:29.610977+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:25:29.610977+0000","last_undegraded":"2023-10-03T21:25:29.610977+0000","last_fullsized":"2023-10-03T21:25:29.610977+0000","mapping_epoch":12,"log_start":"91'213908","ondisk_log_start":"91'213908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":597,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 69s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4958069120,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":223268,"num_read_kb":873326,"num_write":222404,"num_write_kb":10284719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":267,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:11.903712+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:25:11.903712+0000","last_peered":"2023-10-03T21:25:11.903712+0000","last_clean":"2023-10-03T21:25:11.903712+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:25:11.903712+0000","last_undegraded":"2023-10-03T21:25:11.903712+0000","last_fullsized":"2023-10-03T21:25:11.903712+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602511852,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":11,"num_read":80308,"num_read_kb":219334,"num_write":61066,"num_write_kb":200591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63311,"num_flush":15894,"num_flush_kb":63980773,"num_evict":10330,"num_evict_kb":41658830,"num_promote":10366,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1217814528,"data_stored":1217532861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40244,"ondisk_log_size":40244,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":39956241552,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1781958,"num_read_kb":6961252,"num_write":1757781,"num_write_kb":70797563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42589827072,"data_stored":48265730336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80580,"ondisk_log_size":80580,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9547164,"kb_used_data":8339796,"kb_used_omap":376,"kb_used_meta":1206983,"kb_avail":84824676,"statfs":{"total":96636764160,"available":86860468224,"internally_reserved":0,"allocated":8539951104,"data_stored":9716829292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1235951517},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,6,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.86699999999999999,"15min":1.008},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":3.2709999999999999,"5min":5.5670000000000002,"15min":19.114000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.89600000000000002,"15min":1.006},"min":{"1min":0.21199999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":15.244},"last":0.27100000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:25:15 2023","interfaces":[{"interface":"back","average":{"1min":1.9450000000000001,"5min":1.1830000000000001,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":9.0120000000000005,"5min":9.0120000000000005,"15min":30.677},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.077,"15min":1.365},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":6.3289999999999997,"5min":14.073,"15min":30.045999999999999},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739224,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22907208,"kb_used_data":21082616,"kb_used_omap":961,"kb_used_meta":1823614,"kb_avail":71464632,"statfs":{"total":96636764160,"available":73179783168,"internally_reserved":0,"allocated":21588598784,"data_stored":24426178404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1867380773},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,15,12,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:24:34 2023","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":1.266,"15min":2.21},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.153},"max":{"1min":11.055999999999999,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.075,"5min":1.3180000000000001,"15min":1.3089999999999999},"min":{"1min":0.26000000000000001,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":3.8029999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:24:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.472,"15min":1.3080000000000001},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":8.3230000000000004,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.879,"5min":1.3740000000000001,"15min":1.212},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":3.7629999999999999,"5min":14.201000000000001,"15min":14.201000000000001},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15181280,"kb_used_data":13357208,"kb_used_omap":597,"kb_used_meta":1823466,"kb_avail":79190560,"statfs":{"total":96636764160,"available":81091133440,"internally_reserved":0,"allocated":13677780992,"data_stored":15337780838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1867230113},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,9,2,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:24:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85799999999999998,"15min":1.224},"min":{"1min":0.182,"5min":0.17999999999999999,"15min":0.13900000000000001},"max":{"1min":5.9390000000000001,"5min":5.9390000000000001,"15min":21.391999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.982,"5min":1.0740000000000001,"15min":1.123},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":23.422000000000001,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.52400000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:24:33 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.052,"15min":1.119},"min":{"1min":0.222,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":6.0940000000000003,"5min":8.8559999999999999,"15min":10.914999999999999},"last":3.984},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97399999999999998,"15min":0.88400000000000001},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.189},"max":{"1min":5.9900000000000002,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.28199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13371064320,"data_stored":15031551912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21294710784,"data_stored":24132705424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7924051968,"data_stored":9101473000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305426432,"data_stored":305356523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297213952,"data_stored":297143041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":615174144,"data_stored":615033297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:25:33.988 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:33.988 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:33.988 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:33.988 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:33.988 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:33.989 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:25:34.611 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:34.604+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:25:34.624 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:34.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:25:34.624 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:34.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:25:34.624 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:34.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:25:34.624 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:34.616+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:25:35.812 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:35.808+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:25:39.695 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:39.688+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:25:39.717 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:39.712+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:25:39.717 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:39.712+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:25:39.718 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:39.712+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:25:39.718 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:39.712+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:25:39.718 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:25:39.712+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:25:53.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:25:54.764 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:25:54.765 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:25:54.802 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2023-10-03T21:25:54.201491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40613734665,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14792,"num_whiteouts":11,"num_read":1870358,"num_read_kb":7212277,"num_write":1826895,"num_write_kb":71296797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63420,"num_flush":15956,"num_flush_kb":64234113,"num_evict":10379,"num_evict_kb":41859014,"num_promote":10417,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120557,"ondisk_log_size":120557,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":47892892,"kb_used_data":42884220,"kb_used_omap":1935,"kb_used_meta":5006704,"kb_avail":235222628,"statfs":{"total":289910292480,"available":240867971072,"internally_reserved":0,"allocated":43913441280,"data_stored":49565789068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":5126865763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,28,11,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20856856,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":4263,"num_read_kb":16900,"num_write":4242,"num_write_kb":176018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":24,"num_flush":38,"num_flush_kb":155292,"num_evict":27,"num_evict_kb":110360,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002151"},"pg_stats":[{"pgid":"2.7","version":"95'215804","reported_seq":432700,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:50.068621+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:25:50.068621+0000","last_peered":"2023-10-03T21:25:50.068621+0000","last_clean":"2023-10-03T21:25:50.068621+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:25:50.068621+0000","last_undegraded":"2023-10-03T21:25:50.068621+0000","last_fullsized":"2023-10-03T21:25:50.068621+0000","mapping_epoch":12,"log_start":"91'205713","ondisk_log_start":"91'205713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":4999250254,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":216626,"num_read_kb":846810,"num_write":214645,"num_write_kb":10287674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'216315","reported_seq":435688,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:50.095362+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:25:50.095362+0000","last_peered":"2023-10-03T21:25:50.095362+0000","last_clean":"2023-10-03T21:25:50.095362+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:25:50.095362+0000","last_undegraded":"2023-10-03T21:25:50.095362+0000","last_fullsized":"2023-10-03T21:25:50.095362+0000","mapping_epoch":12,"log_start":"91'206313","ondisk_log_start":"91'206313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":296,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5043040575,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":219433,"num_read_kb":854116,"num_write":214929,"num_write_kb":9699496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'234831","reported_seq":471571,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:50.067798+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:25:50.067798+0000","last_peered":"2023-10-03T21:25:50.067798+0000","last_clean":"2023-10-03T21:25:50.067798+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:25:50.067798+0000","last_undegraded":"2023-10-03T21:25:50.067798+0000","last_fullsized":"2023-10-03T21:25:50.067798+0000","mapping_epoch":12,"log_start":"91'224808","ondisk_log_start":"91'224808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5240713719,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":236499,"num_read_kb":925645,"num_write":233585,"num_write_kb":9255857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'224846","reported_seq":451650,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:50.064022+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:25:50.064022+0000","last_peered":"2023-10-03T21:25:50.064022+0000","last_clean":"2023-10-03T21:25:50.064022+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:25:50.064022+0000","last_undegraded":"2023-10-03T21:25:50.064022+0000","last_fullsized":"2023-10-03T21:25:50.064022+0000","mapping_epoch":12,"log_start":"91'214814","ondisk_log_start":"91'214814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5031145955,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":226317,"num_read_kb":883126,"num_write":222316,"num_write_kb":8496289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'34577","reported_seq":56922,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:51.926863+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:25:51.926863+0000","last_peered":"2023-10-03T21:25:51.926863+0000","last_clean":"2023-10-03T21:25:51.926863+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:25:51.926863+0000","last_undegraded":"2023-10-03T21:25:51.926863+0000","last_fullsized":"2023-10-03T21:25:51.926863+0000","mapping_epoch":15,"log_start":"84'24528","ondisk_log_start":"84'24528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":171482546,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":18948,"num_read_kb":55491,"num_write":15986,"num_write_kb":53180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15728,"num_flush":4693,"num_flush_kb":18937208,"num_evict":2553,"num_evict_kb":10317641,"num_promote":2564,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'216719","reported_seq":434380,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:52.062476+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:25:52.062476+0000","last_peered":"2023-10-03T21:25:52.062476+0000","last_clean":"2023-10-03T21:25:52.062476+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:25:52.062476+0000","last_undegraded":"2023-10-03T21:25:52.062476+0000","last_fullsized":"2023-10-03T21:25:52.062476+0000","mapping_epoch":12,"log_start":"91'206711","ondisk_log_start":"91'206711","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4756640065,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":217220,"num_read_kb":848049,"num_write":214350,"num_write_kb":7374856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'34416","reported_seq":56012,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:49.547597+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:25:49.547597+0000","last_peered":"2023-10-03T21:25:49.547597+0000","last_clean":"2023-10-03T21:25:49.547597+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:25:49.547597+0000","last_undegraded":"2023-10-03T21:25:49.547597+0000","last_fullsized":"2023-10-03T21:25:49.547597+0000","mapping_epoch":15,"log_start":"84'24332","ondisk_log_start":"84'24332","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":142196331,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":19768,"num_read_kb":52958,"num_write":14894,"num_write_kb":48301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15778,"num_flush":3511,"num_flush_kb":14105092,"num_evict":2662,"num_evict_kb":10708612,"num_promote":2672,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'227752","reported_seq":456673,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:52.992148+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:25:52.992148+0000","last_peered":"2023-10-03T21:25:52.992148+0000","last_clean":"2023-10-03T21:25:52.992148+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:25:52.992148+0000","last_undegraded":"2023-10-03T21:25:52.992148+0000","last_fullsized":"2023-10-03T21:25:52.992148+0000","mapping_epoch":12,"log_start":"91'217715","ondisk_log_start":"91'217715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5134025125,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":228484,"num_read_kb":893626,"num_write":225186,"num_write_kb":8294767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'30313","reported_seq":50395,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:51.609568+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:25:51.609568+0000","last_peered":"2023-10-03T21:25:51.609568+0000","last_clean":"2023-10-03T21:25:51.609568+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:25:51.609568+0000","last_undegraded":"2023-10-03T21:25:51.609568+0000","last_fullsized":"2023-10-03T21:25:51.609568+0000","mapping_epoch":15,"log_start":"84'20234","ondisk_log_start":"84'20234","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150634851,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16889,"num_read_kb":53028,"num_write":15245,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16488,"num_flush":3909,"num_flush_kb":15773176,"num_evict":2598,"num_evict_kb":10516901,"num_promote":2608,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'218819","reported_seq":440285,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:52.957046+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:25:52.957046+0000","last_peered":"2023-10-03T21:25:52.957046+0000","last_clean":"2023-10-03T21:25:52.957046+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:25:52.957046+0000","last_undegraded":"2023-10-03T21:25:52.957046+0000","last_fullsized":"2023-10-03T21:25:52.957046+0000","mapping_epoch":12,"log_start":"91'208814","ondisk_log_start":"91'208814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":829,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 115s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4831793603,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":220993,"num_read_kb":863790,"num_write":217087,"num_write_kb":7330757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'42732","reported_seq":69154,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:49.698802+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:25:49.698802+0000","last_peered":"2023-10-03T21:25:49.698802+0000","last_clean":"2023-10-03T21:25:49.698802+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:25:49.698802+0000","last_undegraded":"2023-10-03T21:25:49.698802+0000","last_fullsized":"2023-10-03T21:25:49.698802+0000","mapping_epoch":15,"log_start":"84'32731","ondisk_log_start":"84'32731","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":146566361,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":24901,"num_read_kb":58649,"num_write":15193,"num_write_kb":49670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15426,"num_flush":3843,"num_flush_kb":15418637,"num_evict":2566,"num_evict_kb":10315860,"num_promote":2573,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'225056","reported_seq":449613,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:25:49.971562+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:25:49.971562+0000","last_peered":"2023-10-03T21:25:49.971562+0000","last_clean":"2023-10-03T21:25:49.971562+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:25:49.971562+0000","last_undegraded":"2023-10-03T21:25:49.971562+0000","last_fullsized":"2023-10-03T21:25:49.971562+0000","mapping_epoch":12,"log_start":"91'215009","ondisk_log_start":"91'215009","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":775,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 90s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4965654912,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":224174,"num_read_kb":876898,"num_write":223342,"num_write_kb":10353811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":267,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:11.903712+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:25:11.903712+0000","last_peered":"2023-10-03T21:25:11.903712+0000","last_clean":"2023-10-03T21:25:11.903712+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:25:11.903712+0000","last_undegraded":"2023-10-03T21:25:11.903712+0000","last_fullsized":"2023-10-03T21:25:11.903712+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610880089,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":11,"num_read":80506,"num_read_kb":220126,"num_write":61318,"num_write_kb":201396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63420,"num_flush":15956,"num_flush_kb":64234113,"num_evict":10379,"num_evict_kb":41859014,"num_promote":10417,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1242902528,"data_stored":1242621203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40002264208,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1789746,"num_read_kb":6992060,"num_write":1765440,"num_write_kb":71093507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42682454016,"data_stored":48335452448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80245,"ondisk_log_size":80245,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739238,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9578260,"kb_used_data":8352844,"kb_used_omap":376,"kb_used_meta":1225031,"kb_avail":84793580,"statfs":{"total":96636764160,"available":86828625920,"internally_reserved":0,"allocated":8553312256,"data_stored":9725087061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1254432669},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.86699999999999999,"15min":1.008},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":3.2709999999999999,"5min":5.5670000000000002,"15min":19.114000000000001},"last":3.4820000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.89600000000000002,"15min":1.006},"min":{"1min":0.21199999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":15.244},"last":3.3900000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:25:15 2023","interfaces":[{"interface":"back","average":{"1min":1.9450000000000001,"5min":1.1830000000000001,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":9.0120000000000005,"5min":9.0120000000000005,"15min":30.677},"last":3.7160000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.077,"15min":1.365},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":6.3289999999999997,"5min":14.073,"15min":30.045999999999999},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739228,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23052496,"kb_used_data":21129416,"kb_used_omap":961,"kb_used_meta":1922110,"kb_avail":71319344,"statfs":{"total":96636764160,"available":73031008256,"internally_reserved":0,"allocated":21636521984,"data_stored":24463468455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":1968240677},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,16,6,11,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.3779999999999999,"15min":2.1819999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":6.0279999999999996,"5min":11.055999999999999,"15min":28.088999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.2450000000000001,"15min":1.2909999999999999},"min":{"1min":0.19400000000000001,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":7.3799999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":4.4989999999999997}]},{"osd":2,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.391,"15min":1.3},"min":{"1min":0.221,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":12.750999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":4.4569999999999999},{"interface":"front","average":{"1min":1.431,"5min":1.2330000000000001,"15min":1.248},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":11.831,"5min":11.831,"15min":14.201000000000001},"last":0.22}]}]},{"osd":0,"up_from":8,"seq":34359739241,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15262136,"kb_used_data":13401960,"kb_used_omap":597,"kb_used_meta":1859562,"kb_avail":79109704,"statfs":{"total":96636764160,"available":81008336896,"internally_reserved":0,"allocated":13723607040,"data_stored":15377233552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1904192417},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,7,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.01,"15min":1.258},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.156},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":21.391999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.1850000000000001,"15min":1.1679999999999999},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":7.2130000000000001,"5min":23.422000000000001,"15min":23.422000000000001},"last":1.0860000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.252,"15min":1.1910000000000001},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":10.914999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.2589999999999999,"15min":0.96499999999999997},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.189},"max":{"1min":8.9160000000000004,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13400223744,"data_stored":15054325672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21340491776,"data_stored":24167107728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7941738496,"data_stored":9114019048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322179072,"data_stored":322109205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288829440,"data_stored":288758596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":631894016,"data_stored":631753402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:25:54.804 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:54.805 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:54.805 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:54.805 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:54.805 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:25:54.805 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:25:55.597 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:55.591+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:25:55.597 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:55.591+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:25:57.733 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:57.723+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:25:57.734 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:25:57.723+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:26:05.405 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.399+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:26:05.406 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.399+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:26:05.406 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.399+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:26:05.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.407+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:26:05.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.407+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:26:05.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.407+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:26:05.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.407+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:26:05.419 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.411+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:26:05.420 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.415+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:26:05.420 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.415+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:26:05.420 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.415+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:26:05.421 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.415+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:26:05.436 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.431+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:26:05.890 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.879+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:26:05.913 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:26:05.907+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:26:14.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:26:15.506 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:26:15.507 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:26:15.595 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2280,"stamp":"2023-10-03T21:26:14.206743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40671250788,"num_objects":14925,"num_object_clones":9472,"num_object_copies":29850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1877850,"num_read_kb":7241937,"num_write":1834264,"num_write_kb":71579125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63511,"num_flush":16020,"num_flush_kb":64495525,"num_evict":10423,"num_evict_kb":42038730,"num_promote":10464,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120421,"ondisk_log_size":120421,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48109672,"kb_used_data":42992772,"kb_used_omap":1935,"kb_used_meta":5114928,"kb_avail":235005848,"statfs":{"total":289910292480,"available":240645988352,"internally_reserved":0,"allocated":44024598528,"data_stored":49660230666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":5237687139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,7,12,14,16,23,19,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51945473,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4269,"num_read_kb":16904,"num_write":4175,"num_write_kb":130922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":1,"num_flush":29,"num_flush_kb":118496,"num_evict":14,"num_evict_kb":57244,"num_promote":22,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003943"},"pg_stats":[{"pgid":"2.7","version":"95'216641","reported_seq":434398,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:10.508046+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:26:10.508046+0000","last_peered":"2023-10-03T21:26:10.508046+0000","last_clean":"2023-10-03T21:26:10.508046+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:26:10.508046+0000","last_undegraded":"2023-10-03T21:26:10.508046+0000","last_fullsized":"2023-10-03T21:26:10.508046+0000","mapping_epoch":12,"log_start":"91'206613","ondisk_log_start":"91'206613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5012529486,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":217482,"num_read_kb":850202,"num_write":215482,"num_write_kb":10356326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'217323","reported_seq":437623,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:10.528832+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:26:10.528832+0000","last_peered":"2023-10-03T21:26:10.528832+0000","last_clean":"2023-10-03T21:26:10.528832+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:26:10.528832+0000","last_undegraded":"2023-10-03T21:26:10.528832+0000","last_fullsized":"2023-10-03T21:26:10.528832+0000","mapping_epoch":12,"log_start":"91'207314","ondisk_log_start":"91'207314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":510,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 53s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5044429119,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":220339,"num_read_kb":857704,"num_write":215785,"num_write_kb":9715136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'235784","reported_seq":473432,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:10.461763+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:26:10.461763+0000","last_peered":"2023-10-03T21:26:10.461763+0000","last_clean":"2023-10-03T21:26:10.461763+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:26:10.461763+0000","last_undegraded":"2023-10-03T21:26:10.461763+0000","last_fullsized":"2023-10-03T21:26:10.461763+0000","mapping_epoch":12,"log_start":"91'225708","ondisk_log_start":"91'225708","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5248213495,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":237397,"num_read_kb":929193,"num_write":234538,"num_write_kb":9300481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'225815","reported_seq":453495,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:10.491150+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:26:10.491150+0000","last_peered":"2023-10-03T21:26:10.491150+0000","last_clean":"2023-10-03T21:26:10.491150+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:26:10.491150+0000","last_undegraded":"2023-10-03T21:26:10.491150+0000","last_fullsized":"2023-10-03T21:26:10.491150+0000","mapping_epoch":12,"log_start":"91'215814","ondisk_log_start":"91'215814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5035823587,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":227188,"num_read_kb":886570,"num_write":223285,"num_write_kb":8536849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'34745","reported_seq":57182,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:11.528112+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:26:11.528112+0000","last_peered":"2023-10-03T21:26:11.528112+0000","last_clean":"2023-10-03T21:26:11.528112+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:26:11.528112+0000","last_undegraded":"2023-10-03T21:26:11.528112+0000","last_fullsized":"2023-10-03T21:26:11.528112+0000","mapping_epoch":15,"log_start":"84'24728","ondisk_log_start":"84'24728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163147187,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":18998,"num_read_kb":55691,"num_write":16061,"num_write_kb":53426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15729,"num_flush":4720,"num_flush_kb":19047576,"num_evict":2566,"num_evict_kb":10370773,"num_promote":2575,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'217521","reported_seq":436087,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:12.077343+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:26:12.077343+0000","last_peered":"2023-10-03T21:26:12.077343+0000","last_clean":"2023-10-03T21:26:12.077343+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:26:12.077343+0000","last_undegraded":"2023-10-03T21:26:12.077343+0000","last_fullsized":"2023-10-03T21:26:12.077343+0000","mapping_epoch":12,"log_start":"91'207511","ondisk_log_start":"91'207511","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4764275009,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":218118,"num_read_kb":851593,"num_write":215152,"num_write_kb":7406680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'34551","reported_seq":56235,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:10.031807+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:26:10.031807+0000","last_peered":"2023-10-03T21:26:10.031807+0000","last_clean":"2023-10-03T21:26:10.031807+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:26:10.031807+0000","last_undegraded":"2023-10-03T21:26:10.031807+0000","last_fullsized":"2023-10-03T21:26:10.031807+0000","mapping_epoch":15,"log_start":"84'24532","ondisk_log_start":"84'24532","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150589101,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19826,"num_read_kb":53190,"num_write":14960,"num_write_kb":48523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15844,"num_flush":3522,"num_flush_kb":14149936,"num_evict":2672,"num_evict_kb":10749380,"num_promote":2684,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"95'228675","reported_seq":458557,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:13.133427+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:26:13.133427+0000","last_peered":"2023-10-03T21:26:13.133427+0000","last_clean":"2023-10-03T21:26:13.133427+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:26:13.133427+0000","last_undegraded":"2023-10-03T21:26:13.133427+0000","last_fullsized":"2023-10-03T21:26:13.133427+0000","mapping_epoch":12,"log_start":"92'218615","ondisk_log_start":"92'218615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5141610917,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":229440,"num_read_kb":897418,"num_write":226109,"num_write_kb":8322907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'30442","reported_seq":50592,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:11.984969+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:26:11.984969+0000","last_peered":"2023-10-03T21:26:11.984969+0000","last_clean":"2023-10-03T21:26:11.984969+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:26:11.984969+0000","last_undegraded":"2023-10-03T21:26:11.984969+0000","last_fullsized":"2023-10-03T21:26:11.984969+0000","mapping_epoch":15,"log_start":"84'20435","ondisk_log_start":"84'20435","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150647132,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":16928,"num_read_kb":53180,"num_write":15302,"num_write_kb":50427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16481,"num_flush":3927,"num_flush_kb":15846688,"num_evict":2609,"num_evict_kb":10561849,"num_promote":2619,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'219824","reported_seq":442252,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:13.110924+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:26:13.110924+0000","last_peered":"2023-10-03T21:26:13.110924+0000","last_clean":"2023-10-03T21:26:13.110924+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:26:13.110924+0000","last_undegraded":"2023-10-03T21:26:13.110924+0000","last_fullsized":"2023-10-03T21:26:13.110924+0000","mapping_epoch":12,"log_start":"91'209814","ondisk_log_start":"91'209814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":978,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 135s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4833345987,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":221935,"num_read_kb":867522,"num_write":217985,"num_write_kb":7334349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'42890","reported_seq":69406,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:26:10.195325+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:26:10.195325+0000","last_peered":"2023-10-03T21:26:10.195325+0000","last_clean":"2023-10-03T21:26:10.195325+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:26:10.195325+0000","last_undegraded":"2023-10-03T21:26:10.195325+0000","last_fullsized":"2023-10-03T21:26:10.195325+0000","mapping_epoch":15,"log_start":"84'32831","ondisk_log_start":"84'32831","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":158997752,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":24956,"num_read_kb":58869,"num_write":15257,"num_write_kb":49884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15457,"num_flush":3851,"num_flush_kb":15451325,"num_evict":2576,"num_evict_kb":10356728,"num_promote":2586,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'226035","reported_seq":451575,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:10.499204+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:26:10.499204+0000","last_peered":"2023-10-03T21:26:10.499204+0000","last_clean":"2023-10-03T21:26:10.499204+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:26:10.499204+0000","last_undegraded":"2023-10-03T21:26:10.499204+0000","last_fullsized":"2023-10-03T21:26:10.499204+0000","mapping_epoch":12,"log_start":"91'216009","ondisk_log_start":"91'216009","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":936,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 110s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4967051648,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":225137,"num_read_kb":880714,"num_write":224211,"num_write_kb":10402243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":267,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-03T21:25:11.903712+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:25:11.903712+0000","last_peered":"2023-10-03T21:25:11.903712+0000","last_clean":"2023-10-03T21:25:11.903712+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:25:11.903712+0000","last_undegraded":"2023-10-03T21:25:11.903712+0000","last_fullsized":"2023-10-03T21:25:11.903712+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":623381172,"num_objects":230,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":80708,"num_read_kb":220930,"num_write":61580,"num_write_kb":202260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63511,"num_flush":16020,"num_flush_kb":64495525,"num_evict":10423,"num_evict_kb":42038730,"num_promote":10464,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1255374848,"data_stored":1255093649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40102,"ondisk_log_size":40102,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40047279248,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1797036,"num_read_kb":7020916,"num_write":1772547,"num_write_kb":71374971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42773471232,"data_stored":48410286368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80220,"ondisk_log_size":80220,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9645372,"kb_used_data":8383860,"kb_used_omap":376,"kb_used_meta":1261127,"kb_avail":84726468,"statfs":{"total":96636764160,"available":86759903232,"internally_reserved":0,"allocated":8585072640,"data_stored":9755327864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1291394973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.86699999999999999,"15min":1.008},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":3.2709999999999999,"5min":5.5670000000000002,"15min":19.114000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.89600000000000002,"15min":1.006},"min":{"1min":0.21199999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":15.244},"last":0.26000000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:25:15 2023","interfaces":[{"interface":"back","average":{"1min":1.9450000000000001,"5min":1.1830000000000001,"15min":1.0209999999999999},"min":{"1min":0.26800000000000002,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":9.0120000000000005,"5min":9.0120000000000005,"15min":30.677},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.089,"5min":1.077,"15min":1.365},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":6.3289999999999997,"5min":14.073,"15min":30.045999999999999},"last":0.93999999999999995}]}]},{"osd":1,"up_from":8,"seq":34359739232,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23140332,"kb_used_data":21185368,"kb_used_omap":961,"kb_used_meta":1953982,"kb_avail":71231508,"statfs":{"total":96636764160,"available":72941064192,"internally_reserved":0,"allocated":21693816832,"data_stored":24512165896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":2000877605},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,11,7,10,17,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.3779999999999999,"15min":2.1819999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":6.0279999999999996,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.222},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.2450000000000001,"15min":1.2909999999999999},"min":{"1min":0.19400000000000001,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":7.3799999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":1.9790000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.391,"15min":1.3},"min":{"1min":0.221,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":12.750999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.246},{"interface":"front","average":{"1min":1.431,"5min":1.2330000000000001,"15min":1.248},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":11.831,"5min":11.831,"15min":14.201000000000001},"last":6.3479999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15323968,"kb_used_data":13423544,"kb_used_omap":597,"kb_used_meta":1899818,"kb_avail":79047872,"statfs":{"total":96636764160,"available":80945020928,"internally_reserved":0,"allocated":13745709056,"data_stored":15392736906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1945414561},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,1,6,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.01,"15min":1.258},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.156},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":21.391999999999999},"last":1.373},{"interface":"front","average":{"1min":1.351,"5min":1.1850000000000001,"15min":1.1679999999999999},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":7.2130000000000001,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.252,"15min":1.1910000000000001},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":10.914999999999999},"last":6.0819999999999999},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.2589999999999999,"15min":0.96499999999999997},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.189},"max":{"1min":8.9160000000000004,"5min":11.329000000000001,"15min":11.329000000000001},"last":1.345}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13437947904,"data_stored":15085164456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21383081984,"data_stored":24201874576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7952441344,"data_stored":9123247336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313856000,"data_stored":313786127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309641216,"data_stored":309570469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":631877632,"data_stored":631737053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:26:15.597 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:15.597 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:15.597 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:15.597 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:15.598 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:15.598 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:26:25.132 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:25.123+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:26:25.143 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:25.135+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:26:25.144 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:25.135+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:26:25.156 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:25.151+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:26:25.166 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:25.155+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:26:25.170 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:25.163+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:26:33.314 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.303+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:26:33.315 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.307+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:26:33.316 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.307+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:26:33.316 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.307+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:26:33.323 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.311+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:26:33.323 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.311+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:26:33.323 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:33.311+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:26:35.449 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:35.443+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:26:35.456 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:35.447+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:26:35.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:26:36.403 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:26:36.403 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:26:36.458 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2292,"stamp":"2023-10-03T21:26:36.212861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40708696527,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1886676,"num_read_kb":7276897,"num_write":1842938,"num_write_kb":71907586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63618,"num_flush":16088,"num_flush_kb":64773449,"num_evict":10480,"num_evict_kb":42271578,"num_promote":10518,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120647,"ondisk_log_size":120647,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48233300,"kb_used_data":43048468,"kb_used_omap":1935,"kb_used_meta":5178800,"kb_avail":234882220,"statfs":{"total":289910292480,"available":240519393280,"internally_reserved":0,"allocated":44081631232,"data_stored":49682192957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":5303092067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,6,7,26,42,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26767403,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3688,"num_read_kb":14616,"num_write":3652,"num_write_kb":157386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":43,"num_flush":32,"num_flush_kb":130752,"num_evict":26,"num_evict_kb":106288,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003636"},"pg_stats":[{"pgid":"2.7","version":"96'217725","reported_seq":436526,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:36.074940+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:26:36.074940+0000","last_peered":"2023-10-03T21:26:36.074940+0000","last_clean":"2023-10-03T21:26:36.074940+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:26:36.074940+0000","last_undegraded":"2023-10-03T21:26:36.074940+0000","last_fullsized":"2023-10-03T21:26:36.074940+0000","mapping_epoch":12,"log_start":"91'207713","ondisk_log_start":"91'207713","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5026107726,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":218520,"num_read_kb":854314,"num_write":216566,"num_write_kb":10413690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'218580","reported_seq":440069,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:36.077091+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:26:36.077091+0000","last_peered":"2023-10-03T21:26:36.077091+0000","last_clean":"2023-10-03T21:26:36.077091+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:26:36.077091+0000","last_undegraded":"2023-10-03T21:26:36.077091+0000","last_fullsized":"2023-10-03T21:26:36.077091+0000","mapping_epoch":12,"log_start":"92'208514","ondisk_log_start":"92'208514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":723,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 79s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5052248383,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":221492,"num_read_kb":862244,"num_write":216891,"num_write_kb":9756276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'237015","reported_seq":475972,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:36.041702+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:26:36.041702+0000","last_peered":"2023-10-03T21:26:36.041702+0000","last_clean":"2023-10-03T21:26:36.041702+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:26:36.041702+0000","last_undegraded":"2023-10-03T21:26:36.041702+0000","last_fullsized":"2023-10-03T21:26:36.041702+0000","mapping_epoch":12,"log_start":"92'227008","ondisk_log_start":"92'227008","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5256376823,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":238695,"num_read_kb":934333,"num_write":235769,"num_write_kb":9346249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'226949","reported_seq":455805,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:36.067434+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:26:36.067434+0000","last_peered":"2023-10-03T21:26:36.067434+0000","last_clean":"2023-10-03T21:26:36.067434+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:26:36.067434+0000","last_undegraded":"2023-10-03T21:26:36.067434+0000","last_fullsized":"2023-10-03T21:26:36.067434+0000","mapping_epoch":12,"log_start":"92'216914","ondisk_log_start":"92'216914","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5043708387,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":228358,"num_read_kb":891206,"num_write":224419,"num_write_kb":8594449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'34860","reported_seq":57379,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:32.023614+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:26:32.023614+0000","last_peered":"2023-10-03T21:26:32.023614+0000","last_clean":"2023-10-03T21:26:32.023614+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:26:32.023614+0000","last_undegraded":"2023-10-03T21:26:32.023614+0000","last_fullsized":"2023-10-03T21:26:32.023614+0000","mapping_epoch":15,"log_start":"84'24828","ondisk_log_start":"84'24828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":163126709,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":19046,"num_read_kb":55883,"num_write":16116,"num_write_kb":53604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15731,"num_flush":4737,"num_flush_kb":19117008,"num_evict":2576,"num_evict_kb":10411677,"num_promote":2585,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'218401","reported_seq":437823,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:32.204852+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:26:32.204852+0000","last_peered":"2023-10-03T21:26:32.204852+0000","last_clean":"2023-10-03T21:26:32.204852+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:26:32.204852+0000","last_undegraded":"2023-10-03T21:26:32.204852+0000","last_fullsized":"2023-10-03T21:26:32.204852+0000","mapping_epoch":12,"log_start":"92'208311","ondisk_log_start":"92'208311","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4765786433,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":218966,"num_read_kb":854957,"num_write":216032,"num_write_kb":7446996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'34695","reported_seq":56496,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:35.975429+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:26:35.975429+0000","last_peered":"2023-10-03T21:26:35.975429+0000","last_clean":"2023-10-03T21:26:35.975429+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:26:35.975429+0000","last_undegraded":"2023-10-03T21:26:35.975429+0000","last_fullsized":"2023-10-03T21:26:35.975429+0000","mapping_epoch":15,"log_start":"84'24632","ondisk_log_start":"84'24632","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150621900,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19901,"num_read_kb":53490,"num_write":15038,"num_write_kb":48787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15875,"num_flush":3540,"num_flush_kb":14223512,"num_evict":2683,"num_evict_kb":10794264,"num_promote":2695,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"96'229601","reported_seq":460449,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:33.369500+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:26:33.369500+0000","last_peered":"2023-10-03T21:26:33.369500+0000","last_clean":"2023-10-03T21:26:33.369500+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:26:33.369500+0000","last_undegraded":"2023-10-03T21:26:33.369500+0000","last_fullsized":"2023-10-03T21:26:33.369500+0000","mapping_epoch":12,"log_start":"92'219515","ondisk_log_start":"92'219515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5146046885,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":230401,"num_read_kb":901234,"num_write":227035,"num_write_kb":8351123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'30555","reported_seq":50782,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:31.910605+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:26:31.910605+0000","last_peered":"2023-10-03T21:26:31.910605+0000","last_clean":"2023-10-03T21:26:31.910605+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:26:31.910605+0000","last_undegraded":"2023-10-03T21:26:31.910605+0000","last_fullsized":"2023-10-03T21:26:31.910605+0000","mapping_epoch":15,"log_start":"84'20535","ondisk_log_start":"84'20535","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146456947,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":16980,"num_read_kb":53388,"num_write":15363,"num_write_kb":50610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16504,"num_flush":3940,"num_flush_kb":15899836,"num_evict":2624,"num_evict_kb":10623177,"num_promote":2633,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"96'220818","reported_seq":444201,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:33.364495+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:26:33.364495+0000","last_peered":"2023-10-03T21:26:33.364495+0000","last_clean":"2023-10-03T21:26:33.364495+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:26:33.364495+0000","last_undegraded":"2023-10-03T21:26:33.364495+0000","last_fullsized":"2023-10-03T21:26:33.364495+0000","mapping_epoch":12,"log_start":"92'210815","ondisk_log_start":"92'210815","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1078,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 155s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4838015427,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":222872,"num_read_kb":871246,"num_write":218909,"num_write_kb":7350309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'43072","reported_seq":69716,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:35.934527+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:26:35.934527+0000","last_peered":"2023-10-03T21:26:35.934527+0000","last_clean":"2023-10-03T21:26:35.934527+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:26:35.934527+0000","last_undegraded":"2023-10-03T21:26:35.934527+0000","last_fullsized":"2023-10-03T21:26:35.934527+0000","mapping_epoch":15,"log_start":"84'33031","ondisk_log_start":"84'33031","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150687019,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":25023,"num_read_kb":59137,"num_write":15343,"num_write_kb":50140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15508,"num_flush":3871,"num_flush_kb":15533093,"num_evict":2597,"num_evict_kb":10442460,"num_promote":2605,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'227302","reported_seq":454055,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:36.050616+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:26:36.050616+0000","last_peered":"2023-10-03T21:26:36.050616+0000","last_clean":"2023-10-03T21:26:36.050616+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:26:36.050616+0000","last_undegraded":"2023-10-03T21:26:36.050616+0000","last_fullsized":"2023-10-03T21:26:36.050616+0000","mapping_epoch":12,"log_start":"92'217209","ondisk_log_start":"92'217209","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1167,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 136s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4968923520,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":226316,"num_read_kb":885374,"num_write":225320,"num_write_kb":10443459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":268,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-03T21:25:21.808528+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:25:21.808528+0000","last_peered":"2023-10-03T21:25:21.808528+0000","last_clean":"2023-10-03T21:25:21.808528+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:25:21.808528+0000","last_undegraded":"2023-10-03T21:25:21.808528+0000","last_fullsized":"2023-10-03T21:25:21.808528+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610892575,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":80950,"num_read_kb":221898,"num_write":61860,"num_write_kb":203141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63618,"num_flush":16088,"num_flush_kb":64773449,"num_evict":10480,"num_evict_kb":42271578,"num_promote":10518,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1217855488,"data_stored":1217574462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40156,"ondisk_log_size":40156,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40097213584,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1805620,"num_read_kb":7054908,"num_write":1780941,"num_write_kb":71702551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42867200000,"data_stored":48468568352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80392,"ondisk_log_size":80392,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739246,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9631324,"kb_used_data":8373972,"kb_used_omap":376,"kb_used_meta":1256967,"kb_avail":84740516,"statfs":{"total":96636764160,"available":86774288384,"internally_reserved":0,"allocated":8574947328,"data_stored":9738333309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1287135133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89000000000000001,"15min":0.94699999999999995},"min":{"1min":0.191,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":4.3029999999999999,"5min":5.5670000000000002,"15min":8.2720000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.93200000000000005,"15min":0.97399999999999998},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":6.0659999999999998,"5min":7.2220000000000004,"15min":15.244},"last":0.26700000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.1850000000000001,"15min":1.0409999999999999},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.0599999999999996,"5min":9.0120000000000005,"15min":30.677},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0760000000000001,"15min":1.3049999999999999},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.19900000000000001},"max":{"1min":4.7290000000000001,"5min":14.073,"15min":30.045999999999999},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739237,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23231000,"kb_used_data":21226060,"kb_used_omap":961,"kb_used_meta":2003966,"kb_avail":71140840,"statfs":{"total":96636764160,"available":72848220160,"internally_reserved":0,"allocated":21735485440,"data_stored":24534026013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":2052061221},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,15,35,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.3779999999999999,"15min":2.1819999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":6.0279999999999996,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.2450000000000001,"15min":1.2909999999999999},"min":{"1min":0.19400000000000001,"5min":0.189,"15min":0.16600000000000001},"max":{"1min":7.3799999999999999,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.391,"15min":1.3},"min":{"1min":0.221,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":12.750999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.431,"5min":1.2330000000000001,"15min":1.248},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":11.831,"5min":11.831,"15min":14.201000000000001},"last":0.33300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15370976,"kb_used_data":13448436,"kb_used_omap":597,"kb_used_meta":1917866,"kb_avail":79000864,"statfs":{"total":96636764160,"available":80896884736,"internally_reserved":0,"allocated":13771198464,"data_stored":15409833635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1963895713},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,3,3,7,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.01,"15min":1.258},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.156},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":21.391999999999999},"last":0.315},{"interface":"front","average":{"1min":1.351,"5min":1.1850000000000001,"15min":1.1679999999999999},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":7.2130000000000001,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:25:36 2023","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.252,"15min":1.1910000000000001},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":10.914999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.2589999999999999,"15min":0.96499999999999997},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.189},"max":{"1min":8.9160000000000004,"5min":11.329000000000001,"15min":11.329000000000001},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13463486464,"data_stored":15102269352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21440536576,"data_stored":24239172752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7963176960,"data_stored":9127126248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309645312,"data_stored":309575464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301371392,"data_stored":301300727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606838784,"data_stored":606698271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:26:36.865 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:36.866 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:36.866 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:36.866 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:36.867 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:36.869 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:26:36.871 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:36.567+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:26:36.871 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:36.575+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:26:36.871 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:36.575+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:26:36.871 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:36.579+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:26:36.873 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:36.591+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:26:36.873 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:36.591+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:26:39.172 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.167+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:26:39.184 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.175+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:26:39.184 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.175+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:26:39.184 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.175+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:26:39.195 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.187+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:26:39.195 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.187+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:26:39.196 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:26:39.187+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:26:56.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:26:57.660 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:26:57.665 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:26:57.706 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2302,"stamp":"2023-10-03T21:26:56.224997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40729246241,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1892977,"num_read_kb":7301801,"num_write":1849149,"num_write_kb":72213967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63700,"num_flush":16160,"num_flush_kb":65067713,"num_evict":10521,"num_evict_kb":42439246,"num_promote":10558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120717,"ondisk_log_size":120717,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48428012,"kb_used_data":43113612,"kb_used_omap":1935,"kb_used_meta":5312432,"kb_avail":234687508,"statfs":{"total":289910292480,"available":240320008192,"internally_reserved":0,"allocated":44148338688,"data_stored":49738570119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1981597,"internal_metadata":5439931235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,11,16,23,22,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23109674,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3696,"num_read_kb":14608,"num_write":3675,"num_write_kb":181926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":42,"num_flush":44,"num_flush_kb":179748,"num_evict":23,"num_evict_kb":94104,"num_promote":25,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010729"},"pg_stats":[{"pgid":"2.7","version":"96'218376","reported_seq":437809,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.598157+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:26:51.598157+0000","last_peered":"2023-10-03T21:26:51.598157+0000","last_clean":"2023-10-03T21:26:51.598157+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:26:51.598157+0000","last_undegraded":"2023-10-03T21:26:51.598157+0000","last_fullsized":"2023-10-03T21:26:51.598157+0000","mapping_epoch":12,"log_start":"92'208313","ondisk_log_start":"92'208313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5027053902,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":219151,"num_read_kb":856814,"num_write":217217,"num_write_kb":10452950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'219316","reported_seq":441475,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:51.605001+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:26:51.605001+0000","last_peered":"2023-10-03T21:26:51.605001+0000","last_clean":"2023-10-03T21:26:51.605001+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:26:51.605001+0000","last_undegraded":"2023-10-03T21:26:51.605001+0000","last_fullsized":"2023-10-03T21:26:51.605001+0000","mapping_epoch":12,"log_start":"92'209314","ondisk_log_start":"92'209314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":883,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 94s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5056241983,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":222147,"num_read_kb":864836,"num_write":217516,"num_write_kb":9820024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'237717","reported_seq":477350,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.557813+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:26:51.557813+0000","last_peered":"2023-10-03T21:26:51.557813+0000","last_clean":"2023-10-03T21:26:51.557813+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:26:51.557813+0000","last_undegraded":"2023-10-03T21:26:51.557813+0000","last_fullsized":"2023-10-03T21:26:51.557813+0000","mapping_epoch":12,"log_start":"92'227708","ondisk_log_start":"92'227708","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5257650679,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":239372,"num_read_kb":937005,"num_write":236471,"num_write_kb":9353149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'227654","reported_seq":457215,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.564513+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:26:51.564513+0000","last_peered":"2023-10-03T21:26:51.564513+0000","last_clean":"2023-10-03T21:26:51.564513+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:26:51.564513+0000","last_undegraded":"2023-10-03T21:26:51.564513+0000","last_fullsized":"2023-10-03T21:26:51.564513+0000","mapping_epoch":12,"log_start":"92'217614","ondisk_log_start":"92'217614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5044797923,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":229058,"num_read_kb":893990,"num_write":225124,"num_write_kb":8646305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'34993","reported_seq":57595,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.153379+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:26:51.153379+0000","last_peered":"2023-10-03T21:26:51.153379+0000","last_clean":"2023-10-03T21:26:51.153379+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:26:51.153379+0000","last_undegraded":"2023-10-03T21:26:51.153379+0000","last_fullsized":"2023-10-03T21:26:51.153379+0000","mapping_epoch":15,"log_start":"84'24928","ondisk_log_start":"84'24928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158940610,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19098,"num_read_kb":56091,"num_write":16173,"num_write_kb":53798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15744,"num_flush":4757,"num_flush_kb":19198692,"num_evict":2584,"num_evict_kb":10444381,"num_promote":2592,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'219264","reported_seq":439602,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:52.329271+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:26:52.329271+0000","last_peered":"2023-10-03T21:26:52.329271+0000","last_clean":"2023-10-03T21:26:52.329271+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:26:52.329271+0000","last_undegraded":"2023-10-03T21:26:52.329271+0000","last_fullsized":"2023-10-03T21:26:52.329271+0000","mapping_epoch":12,"log_start":"92'209211","ondisk_log_start":"92'209211","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4770271553,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":219876,"num_read_kb":858549,"num_write":216895,"num_write_kb":7466800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'34812","reported_seq":56698,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.508482+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:26:51.508482+0000","last_peered":"2023-10-03T21:26:51.508482+0000","last_clean":"2023-10-03T21:26:51.508482+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:26:51.508482+0000","last_undegraded":"2023-10-03T21:26:51.508482+0000","last_fullsized":"2023-10-03T21:26:51.508482+0000","mapping_epoch":15,"log_start":"84'24732","ondisk_log_start":"84'24732","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146460363,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":19949,"num_read_kb":53682,"num_write":15103,"num_write_kb":49005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15874,"num_flush":3563,"num_flush_kb":14317572,"num_evict":2693,"num_evict_kb":10835160,"num_promote":2704,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"96'230487","reported_seq":462320,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:53.455772+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:26:53.455772+0000","last_peered":"2023-10-03T21:26:53.455772+0000","last_clean":"2023-10-03T21:26:53.455772+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:26:53.455772+0000","last_undegraded":"2023-10-03T21:26:53.455772+0000","last_fullsized":"2023-10-03T21:26:53.455772+0000","mapping_epoch":12,"log_start":"92'220415","ondisk_log_start":"92'220415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5150560677,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":231372,"num_read_kb":905046,"num_write":227921,"num_write_kb":8391379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'30675","reported_seq":50973,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.647772+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:26:51.647772+0000","last_peered":"2023-10-03T21:26:51.647772+0000","last_clean":"2023-10-03T21:26:51.647772+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:26:51.647772+0000","last_undegraded":"2023-10-03T21:26:51.647772+0000","last_fullsized":"2023-10-03T21:26:51.647772+0000","mapping_epoch":15,"log_start":"84'20635","ondisk_log_start":"84'20635","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146489755,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17033,"num_read_kb":53600,"num_write":15421,"num_write_kb":50781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16544,"num_flush":3950,"num_flush_kb":15940680,"num_evict":2639,"num_evict_kb":10684509,"num_promote":2648,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"96'221761","reported_seq":446047,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:53.405853+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:26:53.405853+0000","last_peered":"2023-10-03T21:26:53.405853+0000","last_clean":"2023-10-03T21:26:53.405853+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:26:53.405853+0000","last_undegraded":"2023-10-03T21:26:53.405853+0000","last_fullsized":"2023-10-03T21:26:53.405853+0000","mapping_epoch":12,"log_start":"92'211715","ondisk_log_start":"92'211715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1254,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 175s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4845302211,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":223752,"num_read_kb":874706,"num_write":219735,"num_write_kb":7406721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'43196","reported_seq":69919,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:51.407006+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:26:51.407006+0000","last_peered":"2023-10-03T21:26:51.407006+0000","last_clean":"2023-10-03T21:26:51.407006+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:26:51.407006+0000","last_undegraded":"2023-10-03T21:26:51.407006+0000","last_fullsized":"2023-10-03T21:26:51.407006+0000","mapping_epoch":15,"log_start":"84'33131","ondisk_log_start":"84'33131","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154856777,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25060,"num_read_kb":59285,"num_write":15407,"num_write_kb":50362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15538,"num_flush":3890,"num_flush_kb":15610769,"num_evict":2605,"num_evict_kb":10475196,"num_promote":2614,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'228092","reported_seq":455544,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:26:51.563188+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:26:51.563188+0000","last_peered":"2023-10-03T21:26:51.563188+0000","last_clean":"2023-10-03T21:26:51.563188+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:26:51.563188+0000","last_undegraded":"2023-10-03T21:26:51.563188+0000","last_fullsized":"2023-10-03T21:26:51.563188+0000","mapping_epoch":12,"log_start":"92'218009","ondisk_log_start":"92'218009","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1282,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 151s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4970029440,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":227003,"num_read_kb":888106,"num_write":226029,"num_write_kb":10470799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":269,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:31.834585+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:26:31.834585+0000","last_peered":"2023-10-03T21:26:31.834585+0000","last_clean":"2023-10-03T21:26:31.834585+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:26:31.834585+0000","last_undegraded":"2023-10-03T21:26:31.834585+0000","last_fullsized":"2023-10-03T21:26:31.834585+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606747505,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":81140,"num_read_kb":222658,"num_write":62104,"num_write_kb":203946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63700,"num_flush":16160,"num_flush_kb":65067713,"num_evict":10521,"num_evict_kb":42439246,"num_promote":10558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1234604032,"data_stored":1234323170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40250,"ondisk_log_size":40250,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40121908368,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1811731,"num_read_kb":7079052,"num_write":1786908,"num_write_kb":72008127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42923659264,"data_stored":48515299616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80368,"ondisk_log_size":80368,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9691792,"kb_used_data":8395144,"kb_used_omap":376,"kb_used_meta":1296263,"kb_avail":84680048,"statfs":{"total":96636764160,"available":86712369152,"internally_reserved":0,"allocated":8596627456,"data_stored":9755811023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1327374237},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89000000000000001,"15min":0.94699999999999995},"min":{"1min":0.191,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":4.3029999999999999,"5min":5.5670000000000002,"15min":8.2720000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.93200000000000005,"15min":0.97399999999999998},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":6.0659999999999998,"5min":7.2220000000000004,"15min":15.244},"last":0.35499999999999998}]},{"osd":1,"last update":"Tue Oct 3 21:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.1850000000000001,"15min":1.0409999999999999},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.0599999999999996,"5min":9.0120000000000005,"15min":30.677},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0760000000000001,"15min":1.3049999999999999},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.19900000000000001},"max":{"1min":4.7290000000000001,"5min":14.073,"15min":30.045999999999999},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739240,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23294644,"kb_used_data":21249512,"kb_used_omap":961,"kb_used_meta":2044158,"kb_avail":71077196,"statfs":{"total":96636764160,"available":72783048704,"internally_reserved":0,"allocated":21759500288,"data_stored":24554977053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":985051,"internal_metadata":2093217829},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,12,8,14,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:26:41 2023","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.5349999999999999,"15min":2.23},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":6.7759999999999998,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.20000000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3340000000000001,"15min":1.2310000000000001},"min":{"1min":0.19500000000000001,"5min":0.189,"15min":0.183},"max":{"1min":6.4800000000000004,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.79300000000000004}]},{"osd":2,"last update":"Tue Oct 3 21:26:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.3260000000000001,"15min":1.29},"min":{"1min":0.246,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":4.4569999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.0940000000000001,"15min":1.214},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":6.3479999999999999,"5min":11.831,"15min":14.201000000000001},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15441576,"kb_used_data":13468956,"kb_used_omap":597,"kb_used_meta":1972010,"kb_avail":78930264,"statfs":{"total":96636764160,"available":80824590336,"internally_reserved":0,"allocated":13792210944,"data_stored":15427782043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":2019339169},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,3,14,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:26:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.014,"15min":1.23},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.156},"max":{"1min":4.1589999999999998,"5min":7.4699999999999998,"15min":21.391999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2849999999999999,"15min":1.1970000000000001},"min":{"1min":0.247,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":13.823,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:26:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.173,"15min":1.095},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":6.0819999999999999,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.443},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.151,"15min":1.006},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":7.359,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13485588480,"data_stored":15121622952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21461520384,"data_stored":24257391760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7976550400,"data_stored":9136284904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305496064,"data_stored":305426269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301379584,"data_stored":301308948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":627728384,"data_stored":627587953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:26:57.709 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:57.709 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:57.709 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:57.710 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:57.710 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:26:57.710 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:27:17.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:27:18.497 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:27:18.497 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:27:18.544 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2313,"stamp":"2023-10-03T21:27:18.231028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40775117197,"num_objects":14920,"num_object_clones":9472,"num_object_copies":29840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1901786,"num_read_kb":7336657,"num_write":1857926,"num_write_kb":72604034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63552,"num_flush":16247,"num_flush_kb":65423277,"num_evict":10583,"num_evict_kb":42692550,"num_promote":10619,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120604,"ondisk_log_size":120604,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48003324,"kb_used_data":43207972,"kb_used_omap":1936,"kb_used_meta":4793391,"kb_avail":235112196,"statfs":{"total":289910292480,"available":240754888704,"internally_reserved":0,"allocated":44244963328,"data_stored":49806432371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4908432596},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,4,26,18,23,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9736090,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5199,"num_read_kb":20552,"num_write":5157,"num_write_kb":208264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-102,"num_flush":46,"num_flush_kb":187996,"num_evict":41,"num_evict_kb":167572,"num_promote":37,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"2.7","version":"96'219436","reported_seq":439975,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:17.202409+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:27:17.202409+0000","last_peered":"2023-10-03T21:27:17.202409+0000","last_clean":"2023-10-03T21:27:17.202409+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:27:17.202409+0000","last_undegraded":"2023-10-03T21:27:17.202409+0000","last_fullsized":"2023-10-03T21:27:17.202409+0000","mapping_epoch":12,"log_start":"92'209413","ondisk_log_start":"92'209413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5031616846,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":220250,"num_read_kb":861166,"num_write":218277,"num_write_kb":10502050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'220565","reported_seq":443857,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:17.204016+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:27:17.204016+0000","last_peered":"2023-10-03T21:27:17.204016+0000","last_clean":"2023-10-03T21:27:17.204016+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:27:17.204016+0000","last_undegraded":"2023-10-03T21:27:17.204016+0000","last_fullsized":"2023-10-03T21:27:17.204016+0000","mapping_epoch":12,"log_start":"92'210514","ondisk_log_start":"92'210514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1136,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 120s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5070004543,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":223252,"num_read_kb":869208,"num_write":218582,"num_write_kb":9893732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'238944","reported_seq":479781,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:17.184879+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:27:17.184879+0000","last_peered":"2023-10-03T21:27:17.184879+0000","last_clean":"2023-10-03T21:27:17.184879+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:27:17.184879+0000","last_undegraded":"2023-10-03T21:27:17.184879+0000","last_fullsized":"2023-10-03T21:27:17.184879+0000","mapping_epoch":12,"log_start":"92'228908","ondisk_log_start":"92'228908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5268701687,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":240560,"num_read_kb":941693,"num_write":237698,"num_write_kb":9419301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'228815","reported_seq":459539,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:17.143888+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:27:17.143888+0000","last_peered":"2023-10-03T21:27:17.143888+0000","last_clean":"2023-10-03T21:27:17.143888+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:27:17.143888+0000","last_undegraded":"2023-10-03T21:27:17.143888+0000","last_fullsized":"2023-10-03T21:27:17.143888+0000","mapping_epoch":12,"log_start":"92'218814","ondisk_log_start":"92'218814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5055893987,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":230213,"num_read_kb":898538,"num_write":226285,"num_write_kb":8720297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'35158","reported_seq":57886,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:16.816058+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:27:16.816058+0000","last_peered":"2023-10-03T21:27:16.816058+0000","last_clean":"2023-10-03T21:27:16.816058+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:27:16.816058+0000","last_undegraded":"2023-10-03T21:27:16.816058+0000","last_fullsized":"2023-10-03T21:27:16.816058+0000","mapping_epoch":15,"log_start":"84'25128","ondisk_log_start":"84'25128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154795430,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19183,"num_read_kb":56431,"num_write":16242,"num_write_kb":54032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15716,"num_flush":4779,"num_flush_kb":19288600,"num_evict":2594,"num_evict_kb":10485269,"num_promote":2601,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'220339","reported_seq":441711,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:17.493382+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:27:17.493382+0000","last_peered":"2023-10-03T21:27:17.493382+0000","last_clean":"2023-10-03T21:27:17.493382+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:27:17.493382+0000","last_undegraded":"2023-10-03T21:27:17.493382+0000","last_fullsized":"2023-10-03T21:27:17.493382+0000","mapping_epoch":12,"log_start":"92'210311","ondisk_log_start":"92'210311","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4772127041,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":220905,"num_read_kb":862617,"num_write":217970,"num_write_kb":7499672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'34987","reported_seq":57003,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:16.972703+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:27:16.972703+0000","last_peered":"2023-10-03T21:27:16.972703+0000","last_clean":"2023-10-03T21:27:16.972703+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:27:16.972703+0000","last_undegraded":"2023-10-03T21:27:16.972703+0000","last_fullsized":"2023-10-03T21:27:16.972703+0000","mapping_epoch":15,"log_start":"84'24932","ondisk_log_start":"84'24932","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":142286507,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20030,"num_read_kb":54006,"num_write":15182,"num_write_kb":49243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15842,"num_flush":3587,"num_flush_kb":14415624,"num_evict":2712,"num_evict_kb":10912796,"num_promote":2722,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"96'231365","reported_seq":464128,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:13.701230+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:27:13.701230+0000","last_peered":"2023-10-03T21:27:13.701230+0000","last_clean":"2023-10-03T21:27:13.701230+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:27:13.701230+0000","last_undegraded":"2023-10-03T21:27:13.701230+0000","last_fullsized":"2023-10-03T21:27:13.701230+0000","mapping_epoch":12,"log_start":"92'221315","ondisk_log_start":"92'221315","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5154931109,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":232296,"num_read_kb":908702,"num_write":228799,"num_write_kb":8403071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'30851","reported_seq":51238,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:16.904299+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:27:16.904299+0000","last_peered":"2023-10-03T21:27:16.904299+0000","last_clean":"2023-10-03T21:27:16.904299+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:27:16.904299+0000","last_undegraded":"2023-10-03T21:27:16.904299+0000","last_fullsized":"2023-10-03T21:27:16.904299+0000","mapping_epoch":15,"log_start":"84'20835","ondisk_log_start":"84'20835","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150663542,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17091,"num_read_kb":53832,"num_write":15510,"num_write_kb":51058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16507,"num_flush":3967,"num_flush_kb":16010172,"num_evict":2658,"num_evict_kb":10762117,"num_promote":2668,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"96'222713","reported_seq":447875,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:13.674899+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:27:13.674899+0000","last_peered":"2023-10-03T21:27:13.674899+0000","last_clean":"2023-10-03T21:27:13.674899+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:27:13.674899+0000","last_undegraded":"2023-10-03T21:27:13.674899+0000","last_fullsized":"2023-10-03T21:27:13.674899+0000","mapping_epoch":12,"log_start":"92'212615","ondisk_log_start":"92'212615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1400,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 195s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4846694851,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":224609,"num_read_kb":878094,"num_write":220585,"num_write_kb":7438729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'43409","reported_seq":70255,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:16.845657+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:27:16.845657+0000","last_peered":"2023-10-03T21:27:16.845657+0000","last_clean":"2023-10-03T21:27:16.845657+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:27:16.845657+0000","last_undegraded":"2023-10-03T21:27:16.845657+0000","last_fullsized":"2023-10-03T21:27:16.845657+0000","mapping_epoch":15,"log_start":"84'33331","ondisk_log_start":"84'33331","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154840342,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":25119,"num_read_kb":59521,"num_write":15493,"num_write_kb":50648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15487,"num_flush":3914,"num_flush_kb":15708881,"num_evict":2619,"num_evict_kb":10532368,"num_promote":2628,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'229348","reported_seq":457993,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:17.148201+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:27:17.148201+0000","last_peered":"2023-10-03T21:27:17.148201+0000","last_clean":"2023-10-03T21:27:17.148201+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:27:17.148201+0000","last_undegraded":"2023-10-03T21:27:17.148201+0000","last_fullsized":"2023-10-03T21:27:17.148201+0000","mapping_epoch":12,"log_start":"92'219309","ondisk_log_start":"92'219309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1457,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 177s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4971970944,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":228172,"num_read_kb":892758,"num_write":227166,"num_write_kb":10520307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":269,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:31.834585+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:26:31.834585+0000","last_peered":"2023-10-03T21:26:31.834585+0000","last_clean":"2023-10-03T21:26:31.834585+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:26:31.834585+0000","last_undegraded":"2023-10-03T21:26:31.834585+0000","last_fullsized":"2023-10-03T21:26:31.834585+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602585821,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":81423,"num_read_kb":223790,"num_write":62427,"num_write_kb":204981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63552,"num_flush":16247,"num_flush_kb":65423277,"num_evict":10583,"num_evict_kb":42692550,"num_promote":10619,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197080576,"data_stored":1196799409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40179,"ondisk_log_size":40179,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40171941008,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1820257,"num_read_kb":7112776,"num_write":1795362,"num_write_kb":72397159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43024101376,"data_stored":48587032864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80326,"ondisk_log_size":80326,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739254,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9725612,"kb_used_data":8398436,"kb_used_omap":376,"kb_used_meta":1326791,"kb_avail":84646228,"statfs":{"total":96636764160,"available":86677737472,"internally_reserved":0,"allocated":8599998464,"data_stored":9758125106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1358634909},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.89000000000000001,"15min":0.94699999999999995},"min":{"1min":0.191,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":4.3029999999999999,"5min":5.5670000000000002,"15min":8.2720000000000002},"last":0.309},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.93200000000000005,"15min":0.97399999999999998},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":6.0659999999999998,"5min":7.2220000000000004,"15min":15.244},"last":0.23100000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.1850000000000001,"15min":1.0409999999999999},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.0599999999999996,"5min":9.0120000000000005,"15min":30.677},"last":4.3490000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0760000000000001,"15min":1.3049999999999999},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.19900000000000001},"max":{"1min":4.7290000000000001,"5min":14.073,"15min":30.045999999999999},"last":0.20599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739245,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22767248,"kb_used_data":21299464,"kb_used_omap":963,"kb_used_meta":1466812,"kb_avail":71604592,"statfs":{"total":96636764160,"available":73323102208,"internally_reserved":0,"allocated":21810651136,"data_stored":24591587047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1502015894},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,20,13,18],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:26:41 2023","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.5349999999999999,"15min":2.23},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":6.7759999999999998,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.252},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3340000000000001,"15min":1.2310000000000001},"min":{"1min":0.19500000000000001,"5min":0.189,"15min":0.183},"max":{"1min":6.4800000000000004,"5min":28.699999999999999,"15min":28.699999999999999},"last":4.4980000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:26:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.3260000000000001,"15min":1.29},"min":{"1min":0.246,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":4.4569999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":4.5449999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.0940000000000001,"15min":1.214},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":6.3479999999999999,"5min":11.831,"15min":14.201000000000001},"last":1.052}]}]},{"osd":0,"up_from":8,"seq":34359739258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15510464,"kb_used_data":13510072,"kb_used_omap":597,"kb_used_meta":1999786,"kb_avail":78861376,"statfs":{"total":96636764160,"available":80754049024,"internally_reserved":0,"allocated":13834313728,"data_stored":15456720218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":2047781793},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:26:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.014,"15min":1.23},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.156},"max":{"1min":4.1589999999999998,"5min":7.4699999999999998,"15min":21.391999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2849999999999999,"15min":1.1970000000000001},"min":{"1min":0.247,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":13.823,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Tue Oct 3 21:26:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.173,"15min":1.095},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":6.0819999999999999,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.151,"15min":1.006},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":7.359,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13527736320,"data_stored":15150593960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21512388608,"data_stored":24293776528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7983976448,"data_stored":9142662376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305524736,"data_stored":305454876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301383680,"data_stored":301312961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":590172160,"data_stored":590031572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:27:18.547 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:18.547 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:18.547 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:18.547 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:18.548 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:18.548 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:27:28.947 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:28.942+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:27:28.947 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:28.942+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:27:37.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:37.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:27:37.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:37.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:27:37.846 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:37.842+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:27:37.867 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:37.862+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:27:37.867 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:37.862+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:27:37.868 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:27:37.862+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:27:38.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:27:39.329 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:27:39.329 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:27:39.381 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2323,"stamp":"2023-10-03T21:27:38.242283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40826108205,"num_objects":14923,"num_object_clones":9472,"num_object_copies":29846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14792,"num_whiteouts":11,"num_read":1909053,"num_read_kb":7365385,"num_write":1865038,"num_write_kb":72840483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63456,"num_flush":16305,"num_flush_kb":65660313,"num_evict":10633,"num_evict_kb":42896854,"num_promote":10672,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120745,"ondisk_log_size":120745,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":47636796,"kb_used_data":43314332,"kb_used_omap":1936,"kb_used_meta":4320495,"kb_avail":235478724,"statfs":{"total":289910292480,"available":241130213376,"internally_reserved":0,"allocated":44353875968,"data_stored":49883666397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4424187092},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,20,14,32,16,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20344736,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":4975,"num_read_kb":19668,"num_write":4900,"num_write_kb":186805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-96,"num_flush":48,"num_flush_kb":196148,"num_evict":41,"num_evict_kb":167548,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009672"},"pg_stats":[{"pgid":"2.7","version":"96'220261","reported_seq":441660,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.750340+0000","last_change":"2023-10-03T21:20:16.184099+0000","last_active":"2023-10-03T21:27:37.750340+0000","last_peered":"2023-10-03T21:27:37.750340+0000","last_clean":"2023-10-03T21:27:37.750340+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:27:37.750340+0000","last_undegraded":"2023-10-03T21:27:37.750340+0000","last_fullsized":"2023-10-03T21:27:37.750340+0000","mapping_epoch":12,"log_start":"92'210213","ondisk_log_start":"92'210213","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5041914190,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":221104,"num_read_kb":864550,"num_write":219102,"num_write_kb":10542022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'221574","reported_seq":445699,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:37.762376+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:27:37.762376+0000","last_peered":"2023-10-03T21:27:37.762376+0000","last_clean":"2023-10-03T21:27:37.762376+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:27:37.762376+0000","last_undegraded":"2023-10-03T21:27:37.762376+0000","last_fullsized":"2023-10-03T21:27:37.762376+0000","mapping_epoch":12,"log_start":"93'211514","ondisk_log_start":"93'211514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1384,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 140s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5071319359,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":224069,"num_read_kb":872448,"num_write":219414,"num_write_kb":9925644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'239865","reported_seq":481616,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.747123+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:27:37.747123+0000","last_peered":"2023-10-03T21:27:37.747123+0000","last_clean":"2023-10-03T21:27:37.747123+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:27:37.747123+0000","last_undegraded":"2023-10-03T21:27:37.747123+0000","last_fullsized":"2023-10-03T21:27:37.747123+0000","mapping_epoch":12,"log_start":"93'229808","ondisk_log_start":"93'229808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5270168055,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":241472,"num_read_kb":945301,"num_write":238619,"num_write_kb":9451561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'229662","reported_seq":461314,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.699173+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:27:37.699173+0000","last_peered":"2023-10-03T21:27:37.699173+0000","last_clean":"2023-10-03T21:27:37.699173+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:27:37.699173+0000","last_undegraded":"2023-10-03T21:27:37.699173+0000","last_fullsized":"2023-10-03T21:27:37.699173+0000","mapping_epoch":12,"log_start":"92'219614","ondisk_log_start":"92'219614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5057335779,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":231133,"num_read_kb":902162,"num_write":227132,"num_write_kb":8731865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'35323","reported_seq":58140,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.546959+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:27:37.546959+0000","last_peered":"2023-10-03T21:27:37.546959+0000","last_clean":"2023-10-03T21:27:37.546959+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:27:37.546959+0000","last_undegraded":"2023-10-03T21:27:37.546959+0000","last_fullsized":"2023-10-03T21:27:37.546959+0000","mapping_epoch":15,"log_start":"84'25228","ondisk_log_start":"84'25228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154856846,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":19238,"num_read_kb":56651,"num_write":16317,"num_write_kb":54258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15692,"num_flush":4798,"num_flush_kb":19366228,"num_evict":2612,"num_evict_kb":10558789,"num_promote":2619,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'221155","reported_seq":443380,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.680940+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:27:37.680940+0000","last_peered":"2023-10-03T21:27:37.680940+0000","last_clean":"2023-10-03T21:27:37.680940+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:27:37.680940+0000","last_undegraded":"2023-10-03T21:27:37.680940+0000","last_fullsized":"2023-10-03T21:27:37.680940+0000","mapping_epoch":12,"log_start":"92'211111","ondisk_log_start":"92'211111","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4779594049,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":221746,"num_read_kb":865933,"num_write":218786,"num_write_kb":7523360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'35142","reported_seq":57240,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.652813+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:27:37.652813+0000","last_peered":"2023-10-03T21:27:37.652813+0000","last_clean":"2023-10-03T21:27:37.652813+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:27:37.652813+0000","last_undegraded":"2023-10-03T21:27:37.652813+0000","last_fullsized":"2023-10-03T21:27:37.652813+0000","mapping_epoch":15,"log_start":"84'25133","ondisk_log_start":"84'25133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150625946,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20080,"num_read_kb":54202,"num_write":15249,"num_write_kb":49461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15825,"num_flush":3600,"num_flush_kb":14468824,"num_evict":2724,"num_evict_kb":10961868,"num_promote":2736,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"96'232269","reported_seq":465954,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:33.979595+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:27:33.979595+0000","last_peered":"2023-10-03T21:27:33.979595+0000","last_clean":"2023-10-03T21:27:33.979595+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:27:33.979595+0000","last_undegraded":"2023-10-03T21:27:33.979595+0000","last_fullsized":"2023-10-03T21:27:33.979595+0000","mapping_epoch":12,"log_start":"92'222215","ondisk_log_start":"92'222215","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5162561957,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":233212,"num_read_kb":912342,"num_write":229703,"num_write_kb":8427047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'30968","reported_seq":51434,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.622142+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:27:37.622142+0000","last_peered":"2023-10-03T21:27:37.622142+0000","last_clean":"2023-10-03T21:27:37.622142+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:27:37.622142+0000","last_undegraded":"2023-10-03T21:27:37.622142+0000","last_fullsized":"2023-10-03T21:27:37.622142+0000","mapping_epoch":15,"log_start":"84'20935","ondisk_log_start":"84'20935","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154902869,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17145,"num_read_kb":54048,"num_write":15563,"num_write_kb":51241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16474,"num_flush":3981,"num_flush_kb":16067324,"num_evict":2665,"num_evict_kb":10790677,"num_promote":2676,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"96'223666","reported_seq":449761,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:33.985620+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:27:33.985620+0000","last_peered":"2023-10-03T21:27:33.985620+0000","last_clean":"2023-10-03T21:27:33.985620+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:27:33.985620+0000","last_undegraded":"2023-10-03T21:27:33.985620+0000","last_fullsized":"2023-10-03T21:27:33.985620+0000","mapping_epoch":12,"log_start":"93'213615","ondisk_log_start":"93'213615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1570,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 216s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4851007939,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":225520,"num_read_kb":881698,"num_write":221422,"num_write_kb":7466609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'43594","reported_seq":70524,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:27:37.562347+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:27:37.562347+0000","last_peered":"2023-10-03T21:27:37.562347+0000","last_clean":"2023-10-03T21:27:37.562347+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:27:37.562347+0000","last_undegraded":"2023-10-03T21:27:37.562347+0000","last_fullsized":"2023-10-03T21:27:37.562347+0000","mapping_epoch":15,"log_start":"84'33531","ondisk_log_start":"84'33531","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154770688,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":25160,"num_read_kb":59681,"num_write":15556,"num_write_kb":50846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15465,"num_flush":3926,"num_flush_kb":15757937,"num_evict":2632,"num_evict_kb":10585520,"num_promote":2641,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'230393","reported_seq":459962,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:37.695083+0000","last_change":"2023-10-03T21:24:19.619113+0000","last_active":"2023-10-03T21:27:37.695083+0000","last_peered":"2023-10-03T21:27:37.695083+0000","last_clean":"2023-10-03T21:27:37.695083+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:27:37.695083+0000","last_undegraded":"2023-10-03T21:27:37.695083+0000","last_fullsized":"2023-10-03T21:27:37.695083+0000","mapping_epoch":12,"log_start":"93'220309","ondisk_log_start":"93'220309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1700,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4976460160,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":229068,"num_read_kb":896278,"num_write":228038,"num_write_kb":10564675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":269,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:31.834585+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:26:31.834585+0000","last_peered":"2023-10-03T21:26:31.834585+0000","last_clean":"2023-10-03T21:26:31.834585+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:26:31.834585+0000","last_undegraded":"2023-10-03T21:26:31.834585+0000","last_fullsized":"2023-10-03T21:26:31.834585+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":615156349,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":11,"num_read":81623,"num_read_kb":224582,"num_write":62685,"num_write_kb":205806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63456,"num_flush":16305,"num_flush_kb":65660313,"num_evict":10633,"num_evict_kb":42896854,"num_promote":10672,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1238937600,"data_stored":1238656283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40210361488,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1827324,"num_read_kb":7140712,"num_write":1802216,"num_write_kb":72632783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10154,"num_omap_keys":190,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43110752256,"data_stored":48638933280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80446,"ondisk_log_size":80446,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739258,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9767948,"kb_used_data":8422724,"kb_used_omap":376,"kb_used_meta":1344839,"kb_avail":84603892,"statfs":{"total":96636764160,"available":86634385408,"internally_reserved":0,"allocated":8624869376,"data_stored":9775541244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1377116061},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,10,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":0.93799999999999994,"15min":0.91300000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16700000000000001},"max":{"1min":7.4219999999999997,"5min":7.4219999999999997,"15min":8.2720000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.96999999999999997,"15min":0.96099999999999997},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.1689999999999996,"5min":7.2220000000000004,"15min":15.244},"last":4.9640000000000004}]},{"osd":1,"last update":"Tue Oct 3 21:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.345,"15min":1.0640000000000001},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":4.9119999999999999,"5min":9.0120000000000005,"15min":30.677},"last":5.3319999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1419999999999999,"15min":1.3169999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":6.6859999999999999,"5min":14.073,"15min":30.045999999999999},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739249,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22872212,"kb_used_data":21354888,"kb_used_omap":963,"kb_used_meta":1516348,"kb_avail":71499628,"statfs":{"total":96636764160,"available":73215619072,"internally_reserved":0,"allocated":21867405312,"data_stored":24629081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1552740758},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,7,2,19,14,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:26:41 2023","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.5349999999999999,"15min":2.23},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":6.7759999999999998,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3340000000000001,"15min":1.2310000000000001},"min":{"1min":0.19500000000000001,"5min":0.189,"15min":0.183},"max":{"1min":6.4800000000000004,"5min":28.699999999999999,"15min":28.699999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Tue Oct 3 21:26:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.3260000000000001,"15min":1.29},"min":{"1min":0.246,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":4.4569999999999999,"5min":28.585999999999999,"15min":28.585999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.0940000000000001,"15min":1.214},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":6.3479999999999999,"5min":11.831,"15min":14.201000000000001},"last":0.59399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14996636,"kb_used_data":13536720,"kb_used_omap":597,"kb_used_meta":1459306,"kb_avail":79375204,"statfs":{"total":96636764160,"available":81280208896,"internally_reserved":0,"allocated":13861601280,"data_stored":15479043361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1494330273},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,9,13,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:26:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.014,"15min":1.23},"min":{"1min":0.20200000000000001,"5min":0.182,"15min":0.156},"max":{"1min":4.1589999999999998,"5min":7.4699999999999998,"15min":21.391999999999999},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2849999999999999,"15min":1.1970000000000001},"min":{"1min":0.247,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":13.823,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.254}]},{"osd":2,"last update":"Tue Oct 3 21:26:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.173,"15min":1.095},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":6.0819999999999999,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.151,"15min":1.006},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":7.359,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13549621248,"data_stored":15164438440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21560606720,"data_stored":24322759824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8000524288,"data_stored":9151735016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309829632,"data_stored":309759715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305463296,"data_stored":305392538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":623644672,"data_stored":623504030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:27:39.383 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:39.383 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:39.384 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:39.384 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:39.384 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:27:39.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-03T21:27:40.243 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-03T21:27:40.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T21:27:41.164 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T21:27:41.233 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T21:27:42.159 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T21:27:42.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T21:27:43.098 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T21:27:43.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T21:27:44.018 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T21:27:44.062 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:27:54.046 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:27:54.041+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:27:54.050 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:27:54.045+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:28:04.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:28:04.877 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:28:04.878 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:28:04.916 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2337,"stamp":"2023-10-03T21:28:04.256044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40906217629,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1918310,"num_read_kb":7402069,"num_write":1874066,"num_write_kb":73227187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63312,"num_flush":16389,"num_flush_kb":66003205,"num_evict":10689,"num_evict_kb":43125750,"num_promote":10727,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25450,"num_omap_keys":429,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120851,"ondisk_log_size":120851,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":47909124,"kb_used_data":43456152,"kb_used_omap":1936,"kb_used_meta":4450991,"kb_avail":235206396,"statfs":{"total":289910292480,"available":240851349504,"internally_reserved":0,"allocated":44499099648,"data_stored":49987617380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4557814996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,11,3,21,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47222665,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5224,"num_read_kb":20700,"num_write":5086,"num_write_kb":215850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-119,"num_flush":53,"num_flush_kb":216180,"num_evict":32,"num_evict_kb":130808,"num_promote":32,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005965"},"pg_stats":[{"pgid":"2.7","version":"97'221394","reported_seq":443927,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:28:03.371155+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:28:03.371155+0000","last_peered":"2023-10-03T21:28:03.371155+0000","last_clean":"2023-10-03T21:28:03.371155+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:28:03.371155+0000","last_undegraded":"2023-10-03T21:28:03.371155+0000","last_fullsized":"2023-10-03T21:28:03.371155+0000","mapping_epoch":12,"log_start":"93'211313","ondisk_log_start":"93'211313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":136,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5046387022,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":222214,"num_read_kb":868950,"num_write":220140,"num_write_kb":10570702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'222839","reported_seq":448101,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:28:03.319680+0000","last_change":"2023-10-03T21:25:16.890408+0000","last_active":"2023-10-03T21:28:03.319680+0000","last_peered":"2023-10-03T21:28:03.319680+0000","last_clean":"2023-10-03T21:28:03.319680+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:28:03.319680+0000","last_undegraded":"2023-10-03T21:28:03.319680+0000","last_fullsized":"2023-10-03T21:28:03.319680+0000","mapping_epoch":12,"log_start":"93'212814","ondisk_log_start":"93'212814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1653,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 166s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5072769343,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":225176,"num_read_kb":876836,"num_write":220488,"num_write_kb":9987056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'241004","reported_seq":483919,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:03.316509+0000","last_change":"2023-10-03T21:20:18.244595+0000","last_active":"2023-10-03T21:28:03.316509+0000","last_peered":"2023-10-03T21:28:03.316509+0000","last_clean":"2023-10-03T21:28:03.316509+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:28:03.316509+0000","last_undegraded":"2023-10-03T21:28:03.316509+0000","last_fullsized":"2023-10-03T21:28:03.316509+0000","mapping_epoch":12,"log_start":"93'230908","ondisk_log_start":"93'230908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5290631671,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":242624,"num_read_kb":949857,"num_write":239758,"num_write_kb":9513321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'230835","reported_seq":463659,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:03.307304+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:28:03.307304+0000","last_peered":"2023-10-03T21:28:03.307304+0000","last_clean":"2023-10-03T21:28:03.307304+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:28:03.307304+0000","last_undegraded":"2023-10-03T21:28:03.307304+0000","last_fullsized":"2023-10-03T21:28:03.307304+0000","mapping_epoch":12,"log_start":"93'220814","ondisk_log_start":"93'220814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5068542435,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":232298,"num_read_kb":906790,"num_write":228305,"num_write_kb":8781309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'35439","reported_seq":58334,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:02.703558+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:28:02.703558+0000","last_peered":"2023-10-03T21:28:02.703558+0000","last_clean":"2023-10-03T21:28:02.703558+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:28:02.703558+0000","last_undegraded":"2023-10-03T21:28:02.703558+0000","last_fullsized":"2023-10-03T21:28:02.703558+0000","mapping_epoch":15,"log_start":"84'25428","ondisk_log_start":"84'25428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159124853,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19287,"num_read_kb":56847,"num_write":16382,"num_write_kb":54480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15667,"num_flush":4806,"num_flush_kb":19398960,"num_evict":2621,"num_evict_kb":10595533,"num_promote":2629,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'222238","reported_seq":445490,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:02.838365+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:28:02.838365+0000","last_peered":"2023-10-03T21:28:02.838365+0000","last_clean":"2023-10-03T21:28:02.838365+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:28:02.838365+0000","last_undegraded":"2023-10-03T21:28:02.838365+0000","last_fullsized":"2023-10-03T21:28:02.838365+0000","mapping_epoch":12,"log_start":"93'212211","ondisk_log_start":"93'212211","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4793430337,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":222768,"num_read_kb":869969,"num_write":219869,"num_write_kb":7605100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'35314","reported_seq":57514,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:03.268713+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:28:03.268713+0000","last_peered":"2023-10-03T21:28:03.268713+0000","last_clean":"2023-10-03T21:28:03.268713+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:28:03.268713+0000","last_undegraded":"2023-10-03T21:28:03.268713+0000","last_fullsized":"2023-10-03T21:28:03.268713+0000","mapping_epoch":15,"log_start":"84'25233","ondisk_log_start":"84'25233","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154758776,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20136,"num_read_kb":54426,"num_write":15322,"num_write_kb":49691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15791,"num_flush":3621,"num_flush_kb":14554400,"num_evict":2739,"num_evict_kb":11023172,"num_promote":2752,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'233406","reported_seq":468267,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:27:59.299417+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:27:59.299417+0000","last_peered":"2023-10-03T21:27:59.299417+0000","last_clean":"2023-10-03T21:27:59.299417+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:27:59.299417+0000","last_undegraded":"2023-10-03T21:27:59.299417+0000","last_fullsized":"2023-10-03T21:27:59.299417+0000","mapping_epoch":12,"log_start":"93'223315","ondisk_log_start":"93'223315","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5170356645,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":234379,"num_read_kb":916970,"num_write":230840,"num_write_kb":8468375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'31124","reported_seq":51702,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:02.687761+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:28:02.687761+0000","last_peered":"2023-10-03T21:28:02.687761+0000","last_clean":"2023-10-03T21:28:02.687761+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:28:02.687761+0000","last_undegraded":"2023-10-03T21:28:02.687761+0000","last_fullsized":"2023-10-03T21:28:02.687761+0000","mapping_epoch":15,"log_start":"84'21035","ondisk_log_start":"84'21035","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":138035538,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17220,"num_read_kb":54348,"num_write":15656,"num_write_kb":51527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16471,"num_flush":4003,"num_flush_kb":16157252,"num_evict":2686,"num_evict_kb":10876549,"num_promote":2693,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"97'224811","reported_seq":452049,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:27:59.291298+0000","last_change":"2023-10-03T21:23:57.838184+0000","last_active":"2023-10-03T21:27:59.291298+0000","last_peered":"2023-10-03T21:27:59.291298+0000","last_clean":"2023-10-03T21:27:59.291298+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:27:59.291298+0000","last_undegraded":"2023-10-03T21:27:59.291298+0000","last_fullsized":"2023-10-03T21:27:59.291298+0000","mapping_epoch":12,"log_start":"93'214715","ondisk_log_start":"93'214715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1768,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 241s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4868280771,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":226631,"num_read_kb":886090,"num_write":222429,"num_write_kb":7515501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'43826","reported_seq":70909,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:03.170273+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:28:03.170273+0000","last_peered":"2023-10-03T21:28:03.170273+0000","last_clean":"2023-10-03T21:28:03.170273+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:28:03.170273+0000","last_undegraded":"2023-10-03T21:28:03.170273+0000","last_fullsized":"2023-10-03T21:28:03.170273+0000","mapping_epoch":15,"log_start":"84'33731","ondisk_log_start":"84'33731","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":158907566,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25240,"num_read_kb":59997,"num_write":15642,"num_write_kb":51144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15383,"num_flush":3959,"num_flush_kb":15892593,"num_evict":2643,"num_evict_kb":10630496,"num_promote":2653,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'231548","reported_seq":462299,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:03.322655+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:28:03.322655+0000","last_peered":"2023-10-03T21:28:03.322655+0000","last_clean":"2023-10-03T21:28:03.322655+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:28:03.322655+0000","last_undegraded":"2023-10-03T21:28:03.322655+0000","last_fullsized":"2023-10-03T21:28:03.322655+0000","mapping_epoch":12,"log_start":"93'221509","ondisk_log_start":"93'221509","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4984402304,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":230231,"num_read_kb":900898,"num_write":229098,"num_write_kb":10577087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":270,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:41.144832+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:26:41.144832+0000","last_peered":"2023-10-03T21:26:41.144832+0000","last_clean":"2023-10-03T21:26:41.144832+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:26:41.144832+0000","last_undegraded":"2023-10-03T21:26:41.144832+0000","last_fullsized":"2023-10-03T21:26:41.144832+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610826733,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":81883,"num_read_kb":225618,"num_write":63002,"num_write_kb":206842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63312,"num_flush":16389,"num_flush_kb":66003205,"num_evict":10689,"num_evict_kb":43125750,"num_promote":10727,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1230356480,"data_stored":1230074842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40276,"ondisk_log_size":40276,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40294800528,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1836321,"num_read_kb":7176360,"num_write":1810927,"num_write_kb":73018451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11604,"num_omap_keys":212,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43273945088,"data_stored":48763963680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80476,"ondisk_log_size":80476,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9805984,"kb_used_data":8442712,"kb_used_omap":376,"kb_used_meta":1362887,"kb_avail":84565856,"statfs":{"total":96636764160,"available":86595436544,"internally_reserved":0,"allocated":8645337088,"data_stored":9783908835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1395597213},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":0.93799999999999994,"15min":0.91300000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16700000000000001},"max":{"1min":7.4219999999999997,"5min":7.4219999999999997,"15min":8.2720000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.96999999999999997,"15min":0.96099999999999997},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.1689999999999996,"5min":7.2220000000000004,"15min":15.244},"last":0.28399999999999997}]},{"osd":1,"last update":"Tue Oct 3 21:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.345,"15min":1.0640000000000001},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":4.9119999999999999,"5min":9.0120000000000005,"15min":30.677},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1419999999999999,"15min":1.3169999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":6.6859999999999999,"5min":14.073,"15min":30.045999999999999},"last":2.0870000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739254,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23028976,"kb_used_data":21439444,"kb_used_omap":963,"kb_used_meta":1588540,"kb_avail":71342864,"statfs":{"total":96636764160,"available":73055092736,"internally_reserved":0,"allocated":21953990656,"data_stored":24698316004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1626665366},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,8,1,11,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:27:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.581,"15min":2.0059999999999998},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":7.0410000000000004,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.1890000000000001,"15min":1.214},"min":{"1min":0.215,"5min":0.189,"15min":0.183},"max":{"1min":5.2160000000000002,"5min":7.3799999999999999,"15min":28.699999999999999},"last":6.5819999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:27:42 2023","interfaces":[{"interface":"back","average":{"1min":2.5339999999999998,"5min":1.403,"15min":1.3879999999999999},"min":{"1min":0.253,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":7.3849999999999998,"5min":12.750999999999999,"15min":28.585999999999999},"last":6.625},{"interface":"front","average":{"1min":2.641,"5min":1.387,"15min":1.3200000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":8.0570000000000004,"5min":11.831,"15min":14.201000000000001},"last":0.30499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15074164,"kb_used_data":13573996,"kb_used_omap":597,"kb_used_meta":1499562,"kb_avail":79297676,"statfs":{"total":96636764160,"available":81200820224,"internally_reserved":0,"allocated":13899771904,"data_stored":15505392541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1535552417},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,7,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:27:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.056,"15min":1.196},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.16200000000000001},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":21.391999999999999},"last":1.109},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4450000000000001,"15min":1.222},"min":{"1min":0.254,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":11.349,"5min":23.422000000000001,"15min":23.422000000000001},"last":1.1419999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":1.0309999999999999,"15min":1.0740000000000001},"min":{"1min":0.23400000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":0.92700000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.251},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.996,"15min":1.012},"min":{"1min":0.25600000000000001,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":1.181,"5min":8.9160000000000004,"15min":11.329000000000001},"last":6.633}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13601198080,"data_stored":15207307176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21639045120,"data_stored":24383831184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8033701888,"data_stored":9172825320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297230336,"data_stored":297160391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313733120,"data_stored":313662246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":619393024,"data_stored":619252205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:28:04.918 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:04.918 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:04.919 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:04.919 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:04.919 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:28:04.944 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:04.937+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:04.945 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:04.937+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:28:04.945 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:04.937+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:28:04.949 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:04.945+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:28:04.951 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:04.945+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:28:04.951 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:04.945+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:28:20.410 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:20.405+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:28:20.414 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:20.409+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:28:20.422 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:20.417+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:28:24.454 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:24.449+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:24.480 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:24.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:28:24.480 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:24.473+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:28:24.482 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:24.477+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:28:24.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:24.481+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:28:24.487 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:24.481+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:28:24.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:28:25.782 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:28:25.783 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:28:25.817 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2347,"stamp":"2023-10-03T21:28:24.259607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40953518113,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":1925828,"num_read_kb":7431829,"num_write":1881574,"num_write_kb":73595604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63188,"num_flush":16473,"num_flush_kb":66346321,"num_evict":10740,"num_evict_kb":43334198,"num_promote":10778,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120706,"ondisk_log_size":120706,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48120368,"kb_used_data":43540376,"kb_used_omap":1936,"kb_used_meta":4578031,"kb_avail":234995152,"statfs":{"total":289910292480,"available":240635035648,"internally_reserved":0,"allocated":44585345024,"data_stored":50046341484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4687903956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,12,21,11,13,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32878519,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":5211,"num_read_kb":20640,"num_write":5249,"num_write_kb":245243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-73,"num_flush":64,"num_flush_kb":261432,"num_evict":37,"num_evict_kb":151264,"num_promote":38,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"2.7","version":"97'222385","reported_seq":445838,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:28:23.978853+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:28:23.978853+0000","last_peered":"2023-10-03T21:28:23.978853+0000","last_clean":"2023-10-03T21:28:23.978853+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:28:23.978853+0000","last_undegraded":"2023-10-03T21:28:23.978853+0000","last_fullsized":"2023-10-03T21:28:23.978853+0000","mapping_epoch":12,"log_start":"93'212313","ondisk_log_start":"93'212313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":335,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 34s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5047615822,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":223113,"num_read_kb":872514,"num_write":220997,"num_write_kb":10614954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'223849","reported_seq":450084,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:23.967245+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:28:23.967245+0000","last_peered":"2023-10-03T21:28:23.967245+0000","last_clean":"2023-10-03T21:28:23.967245+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:28:23.967245+0000","last_undegraded":"2023-10-03T21:28:23.967245+0000","last_fullsized":"2023-10-03T21:28:23.967245+0000","mapping_epoch":12,"log_start":"93'213814","ondisk_log_start":"93'213814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5077131583,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":226121,"num_read_kb":880568,"num_write":221392,"num_write_kb":10015184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'242056","reported_seq":485969,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:28:23.967882+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:28:23.967882+0000","last_peered":"2023-10-03T21:28:23.967882+0000","last_clean":"2023-10-03T21:28:23.967882+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:28:23.967882+0000","last_undegraded":"2023-10-03T21:28:23.967882+0000","last_fullsized":"2023-10-03T21:28:23.967882+0000","mapping_epoch":12,"log_start":"93'232008","ondisk_log_start":"93'232008","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":164,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 17s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5301027319,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":243593,"num_read_kb":953669,"num_write":240695,"num_write_kb":9574213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'231754","reported_seq":465533,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:23.944487+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:28:23.944487+0000","last_peered":"2023-10-03T21:28:23.944487+0000","last_clean":"2023-10-03T21:28:23.944487+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:28:23.944487+0000","last_undegraded":"2023-10-03T21:28:23.944487+0000","last_fullsized":"2023-10-03T21:28:23.944487+0000","mapping_epoch":12,"log_start":"93'221714","ondisk_log_start":"93'221714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5070008803,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":233247,"num_read_kb":910562,"num_write":229224,"num_write_kb":8813509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'35591","reported_seq":58580,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:22.950646+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:28:22.950646+0000","last_peered":"2023-10-03T21:28:22.950646+0000","last_clean":"2023-10-03T21:28:22.950646+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:28:22.950646+0000","last_undegraded":"2023-10-03T21:28:22.950646+0000","last_fullsized":"2023-10-03T21:28:22.950646+0000","mapping_epoch":15,"log_start":"84'25528","ondisk_log_start":"84'25528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159063362,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":19345,"num_read_kb":57075,"num_write":16461,"num_write_kb":54750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15616,"num_flush":4827,"num_flush_kb":19484832,"num_evict":2632,"num_evict_kb":10640533,"num_promote":2640,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'223137","reported_seq":447292,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:22.967202+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:28:22.967202+0000","last_peered":"2023-10-03T21:28:22.967202+0000","last_clean":"2023-10-03T21:28:22.967202+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:28:22.967202+0000","last_undegraded":"2023-10-03T21:28:22.967202+0000","last_fullsized":"2023-10-03T21:28:22.967202+0000","mapping_epoch":12,"log_start":"93'213111","ondisk_log_start":"93'213111","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4800901441,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":223665,"num_read_kb":873537,"num_write":220768,"num_write_kb":7669792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'35484","reported_seq":57778,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:23.892670+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:28:23.892670+0000","last_peered":"2023-10-03T21:28:23.892670+0000","last_clean":"2023-10-03T21:28:23.892670+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:28:23.892670+0000","last_undegraded":"2023-10-03T21:28:23.892670+0000","last_fullsized":"2023-10-03T21:28:23.892670+0000","mapping_epoch":15,"log_start":"84'25434","ondisk_log_start":"84'25434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150580829,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":20193,"num_read_kb":54654,"num_write":15395,"num_write_kb":49925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15764,"num_flush":3639,"num_flush_kb":14627896,"num_evict":2753,"num_evict_kb":11080384,"num_promote":2765,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'234306","reported_seq":470083,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:19.398839+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:28:19.398839+0000","last_peered":"2023-10-03T21:28:19.398839+0000","last_clean":"2023-10-03T21:28:19.398839+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:28:19.398839+0000","last_undegraded":"2023-10-03T21:28:19.398839+0000","last_fullsized":"2023-10-03T21:28:19.398839+0000","mapping_epoch":12,"log_start":"93'224215","ondisk_log_start":"93'224215","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5177762213,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":235291,"num_read_kb":920574,"num_write":231740,"num_write_kb":8500471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'31247","reported_seq":51911,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:22.638819+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:28:22.638819+0000","last_peered":"2023-10-03T21:28:22.638819+0000","last_clean":"2023-10-03T21:28:22.638819+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:28:22.638819+0000","last_undegraded":"2023-10-03T21:28:22.638819+0000","last_fullsized":"2023-10-03T21:28:22.638819+0000","mapping_epoch":15,"log_start":"84'21236","ondisk_log_start":"84'21236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154775867,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":3,"num_read":17275,"num_read_kb":54568,"num_write":15715,"num_write_kb":51722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16448,"num_flush":4023,"num_flush_kb":16238956,"num_evict":2696,"num_evict_kb":10917425,"num_promote":2707,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"97'225707","reported_seq":453794,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:19.405325+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:28:19.405325+0000","last_peered":"2023-10-03T21:28:19.405325+0000","last_clean":"2023-10-03T21:28:19.405325+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:28:19.405325+0000","last_undegraded":"2023-10-03T21:28:19.405325+0000","last_fullsized":"2023-10-03T21:28:19.405325+0000","mapping_epoch":12,"log_start":"93'215615","ondisk_log_start":"93'215615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4875850179,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":227465,"num_read_kb":889390,"num_write":223306,"num_write_kb":7563877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'44043","reported_seq":71238,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:23.962823+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:28:23.962823+0000","last_peered":"2023-10-03T21:28:23.962823+0000","last_clean":"2023-10-03T21:28:23.962823+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:28:23.962823+0000","last_undegraded":"2023-10-03T21:28:23.962823+0000","last_fullsized":"2023-10-03T21:28:23.962823+0000","mapping_epoch":15,"log_start":"84'34031","ondisk_log_start":"84'34031","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":146369687,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":25297,"num_read_kb":60225,"num_write":15718,"num_write_kb":51382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15360,"num_flush":3984,"num_flush_kb":15994637,"num_evict":2659,"num_evict_kb":10695856,"num_promote":2666,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'232476","reported_seq":464118,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:23.930720+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:28:23.930720+0000","last_peered":"2023-10-03T21:28:23.930720+0000","last_clean":"2023-10-03T21:28:23.930720+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:28:23.930720+0000","last_undegraded":"2023-10-03T21:28:23.930720+0000","last_fullsized":"2023-10-03T21:28:23.930720+0000","mapping_epoch":12,"log_start":"93'222409","ondisk_log_start":"93'222409","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4991840640,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":231117,"num_read_kb":904402,"num_write":230026,"num_write_kb":10633931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":270,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T21:26:41.144832+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:26:41.144832+0000","last_peered":"2023-10-03T21:26:41.144832+0000","last_clean":"2023-10-03T21:26:41.144832+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:26:41.144832+0000","last_undegraded":"2023-10-03T21:26:41.144832+0000","last_fullsized":"2023-10-03T21:26:41.144832+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610789745,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":82110,"num_read_kb":226522,"num_write":63289,"num_write_kb":207779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63188,"num_flush":16473,"num_flush_kb":66346321,"num_evict":10740,"num_evict_kb":43334198,"num_promote":10778,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1213599744,"data_stored":1213317858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40136,"ondisk_log_size":40136,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40342138000,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1843612,"num_read_kb":7205216,"num_write":1818148,"num_write_kb":73385931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43369000960,"data_stored":48831404320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80471,"ondisk_log_size":80471,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739267,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9873824,"kb_used_data":8460568,"kb_used_omap":376,"kb_used_meta":1412871,"kb_avail":84498016,"statfs":{"total":96636764160,"available":86525968384,"internally_reserved":0,"allocated":8663621632,"data_stored":9794234727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1446780829},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":0.93799999999999994,"15min":0.91300000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16700000000000001},"max":{"1min":7.4219999999999997,"5min":7.4219999999999997,"15min":8.2720000000000002},"last":3.7130000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.96999999999999997,"15min":0.96099999999999997},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.1689999999999996,"5min":7.2220000000000004,"15min":15.244},"last":0.26700000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.345,"15min":1.0640000000000001},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":4.9119999999999999,"5min":9.0120000000000005,"15min":30.677},"last":8.9299999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1419999999999999,"15min":1.3169999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":6.6859999999999999,"5min":14.073,"15min":30.045999999999999},"last":14.381}]}]},{"osd":1,"up_from":8,"seq":34359739258,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23091556,"kb_used_data":21465180,"kb_used_omap":963,"kb_used_meta":1625404,"kb_avail":71280284,"statfs":{"total":96636764160,"available":72991010816,"internally_reserved":0,"allocated":21980344320,"data_stored":24710841522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1664414102},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,11,18,7,10,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:27:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.581,"15min":2.0059999999999998},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":7.0410000000000004,"5min":11.055999999999999,"15min":28.088999999999999},"last":10.156000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.1890000000000001,"15min":1.214},"min":{"1min":0.215,"5min":0.189,"15min":0.183},"max":{"1min":5.2160000000000002,"5min":7.3799999999999999,"15min":28.699999999999999},"last":0.373}]},{"osd":2,"last update":"Tue Oct 3 21:27:42 2023","interfaces":[{"interface":"back","average":{"1min":2.5339999999999998,"5min":1.403,"15min":1.3879999999999999},"min":{"1min":0.253,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":7.3849999999999998,"5min":12.750999999999999,"15min":28.585999999999999},"last":0.496},{"interface":"front","average":{"1min":2.641,"5min":1.387,"15min":1.3200000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":8.0570000000000004,"5min":11.831,"15min":14.201000000000001},"last":0.65600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739271,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15154988,"kb_used_data":13614628,"kb_used_omap":597,"kb_used_meta":1539754,"kb_avail":79216852,"statfs":{"total":96636764160,"available":81118056448,"internally_reserved":0,"allocated":13941379072,"data_stored":15541265235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1576709025},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:27:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.056,"15min":1.196},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.16200000000000001},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":21.391999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4450000000000001,"15min":1.222},"min":{"1min":0.254,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":11.349,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":1.0309999999999999,"15min":1.0740000000000001},"min":{"1min":0.23400000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":0.92700000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.309},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.996,"15min":1.012},"min":{"1min":0.25600000000000001,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":1.181,"5min":8.9160000000000004,"15min":11.329000000000001},"last":2.927}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13626159104,"data_stored":15226537896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21686681600,"data_stored":24417569936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8056160256,"data_stored":9187296488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313909248,"data_stored":313839229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297021440,"data_stored":296950516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602669056,"data_stored":602528113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:28:25.819 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:25.819 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:25.819 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:28:26.395 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:26.389+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:28:26.396 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:26.389+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:28:30.952 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:30.941+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:28:39.187 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:39.180+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:39.187 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:39.180+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:28:39.187 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:39.180+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:28:39.188 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:39.180+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:28:45.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:28:47.097 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:28:47.098 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:28:47.136 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2358,"stamp":"2023-10-03T21:28:46.263760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41002211297,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1934165,"num_read_kb":7464805,"num_write":1889756,"num_write_kb":73917976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63124,"num_flush":16539,"num_flush_kb":66615997,"num_evict":10788,"num_evict_kb":43530274,"num_promote":10826,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120470,"ondisk_log_size":120470,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48328960,"kb_used_data":43652824,"kb_used_omap":1936,"kb_used_meta":4674159,"kb_avail":234786560,"statfs":{"total":289910292480,"available":240421437440,"internally_reserved":0,"allocated":44700491776,"data_stored":50150052076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4786339028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,11,12,16,29,20,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23670754,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":4426,"num_read_kb":17488,"num_write":4360,"num_write_kb":184694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-30,"num_flush":33,"num_flush_kb":134880,"num_evict":29,"num_evict_kb":118480,"num_promote":28,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002449"},"pg_stats":[{"pgid":"2.7","version":"97'223444","reported_seq":447847,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:28:44.213795+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:28:44.213795+0000","last_peered":"2023-10-03T21:28:44.213795+0000","last_clean":"2023-10-03T21:28:44.213795+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:28:44.213795+0000","last_undegraded":"2023-10-03T21:28:44.213795+0000","last_fullsized":"2023-10-03T21:28:44.213795+0000","mapping_epoch":12,"log_start":"93'213413","ondisk_log_start":"93'213413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":463,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 54s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5049094478,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":224044,"num_read_kb":876198,"num_write":221966,"num_write_kb":10655574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'224726","reported_seq":451914,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.183448+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:28:44.183448+0000","last_peered":"2023-10-03T21:28:44.183448+0000","last_clean":"2023-10-03T21:28:44.183448+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:28:44.183448+0000","last_undegraded":"2023-10-03T21:28:44.183448+0000","last_fullsized":"2023-10-03T21:28:44.183448+0000","mapping_epoch":12,"log_start":"94'214714","ondisk_log_start":"94'214714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5090607423,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":227065,"num_read_kb":884292,"num_write":222269,"num_write_kb":10055444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'243136","reported_seq":488038,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:28:44.205421+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:28:44.205421+0000","last_peered":"2023-10-03T21:28:44.205421+0000","last_clean":"2023-10-03T21:28:44.205421+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:28:44.205421+0000","last_undegraded":"2023-10-03T21:28:44.205421+0000","last_fullsized":"2023-10-03T21:28:44.205421+0000","mapping_epoch":12,"log_start":"94'233109","ondisk_log_start":"94'233109","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":338,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 38s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5308473847,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":244566,"num_read_kb":957521,"num_write":241652,"num_write_kb":9614757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'232702","reported_seq":467504,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.205580+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:28:44.205580+0000","last_peered":"2023-10-03T21:28:44.205580+0000","last_clean":"2023-10-03T21:28:44.205580+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:28:44.205580+0000","last_undegraded":"2023-10-03T21:28:44.205580+0000","last_fullsized":"2023-10-03T21:28:44.205580+0000","mapping_epoch":12,"log_start":"93'222614","ondisk_log_start":"93'222614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5071466979,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":234267,"num_read_kb":914590,"num_write":230172,"num_write_kb":8841777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'35745","reported_seq":58817,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:42.723658+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:28:42.723658+0000","last_peered":"2023-10-03T21:28:42.723658+0000","last_clean":"2023-10-03T21:28:42.723658+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:28:42.723658+0000","last_undegraded":"2023-10-03T21:28:42.723658+0000","last_fullsized":"2023-10-03T21:28:42.723658+0000","mapping_epoch":15,"log_start":"84'25728","ondisk_log_start":"84'25728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159063333,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19388,"num_read_kb":57239,"num_write":16534,"num_write_kb":55004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15587,"num_flush":4852,"num_flush_kb":19587056,"num_evict":2641,"num_evict_kb":10677337,"num_promote":2649,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'224020","reported_seq":449118,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:42.992355+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:28:42.992355+0000","last_peered":"2023-10-03T21:28:42.992355+0000","last_clean":"2023-10-03T21:28:42.992355+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:28:42.992355+0000","last_undegraded":"2023-10-03T21:28:42.992355+0000","last_fullsized":"2023-10-03T21:28:42.992355+0000","mapping_epoch":12,"log_start":"93'214011","ondisk_log_start":"93'214011","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4802445633,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":224605,"num_read_kb":877269,"num_write":221651,"num_write_kb":7701844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'35651","reported_seq":58028,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.147644+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:28:44.147644+0000","last_peered":"2023-10-03T21:28:44.147644+0000","last_clean":"2023-10-03T21:28:44.147644+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:28:44.147644+0000","last_undegraded":"2023-10-03T21:28:44.147644+0000","last_fullsized":"2023-10-03T21:28:44.147644+0000","mapping_epoch":15,"log_start":"84'25634","ondisk_log_start":"84'25634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146378321,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20244,"num_read_kb":54858,"num_write":15469,"num_write_kb":50163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15752,"num_flush":3653,"num_flush_kb":14685108,"num_evict":2767,"num_evict_kb":11137620,"num_promote":2778,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'235518","reported_seq":472594,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.562542+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:28:44.562542+0000","last_peered":"2023-10-03T21:28:44.562542+0000","last_clean":"2023-10-03T21:28:44.562542+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:28:44.562542+0000","last_undegraded":"2023-10-03T21:28:44.562542+0000","last_fullsized":"2023-10-03T21:28:44.562542+0000","mapping_epoch":12,"log_start":"93'225515","ondisk_log_start":"93'225515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5182497189,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":236580,"num_read_kb":925670,"num_write":232952,"num_write_kb":8529799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'31375","reported_seq":52131,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:42.682555+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:28:42.682555+0000","last_peered":"2023-10-03T21:28:42.682555+0000","last_clean":"2023-10-03T21:28:42.682555+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:28:42.682555+0000","last_undegraded":"2023-10-03T21:28:42.682555+0000","last_fullsized":"2023-10-03T21:28:42.682555+0000","mapping_epoch":15,"log_start":"84'21336","ondisk_log_start":"84'21336","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146444573,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":17341,"num_read_kb":54832,"num_write":15778,"num_write_kb":51908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16418,"num_flush":4038,"num_flush_kb":16300212,"num_evict":2712,"num_evict_kb":10982817,"num_promote":2721,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"97'226846","reported_seq":456073,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.538677+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:28:44.538677+0000","last_peered":"2023-10-03T21:28:44.538677+0000","last_clean":"2023-10-03T21:28:44.538677+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:28:44.538677+0000","last_undegraded":"2023-10-03T21:28:44.538677+0000","last_fullsized":"2023-10-03T21:28:44.538677+0000","mapping_epoch":12,"log_start":"94'216815","ondisk_log_start":"94'216815","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4892680643,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":228595,"num_read_kb":893854,"num_write":224445,"num_write_kb":7617457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'44238","reported_seq":71529,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:43.877991+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:28:43.877991+0000","last_peered":"2023-10-03T21:28:43.877991+0000","last_clean":"2023-10-03T21:28:43.877991+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:28:43.877991+0000","last_undegraded":"2023-10-03T21:28:43.877991+0000","last_fullsized":"2023-10-03T21:28:43.877991+0000","mapping_epoch":15,"log_start":"84'34231","ondisk_log_start":"84'34231","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":158993566,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":25348,"num_read_kb":60429,"num_write":15782,"num_write_kb":51600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15367,"num_flush":3996,"num_flush_kb":16043621,"num_evict":2668,"num_evict_kb":10732500,"num_promote":2678,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'233399","reported_seq":465947,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.199688+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:28:44.199688+0000","last_peered":"2023-10-03T21:28:44.199688+0000","last_clean":"2023-10-03T21:28:44.199688+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:28:44.199688+0000","last_undegraded":"2023-10-03T21:28:44.199688+0000","last_fullsized":"2023-10-03T21:28:44.199688+0000","mapping_epoch":12,"log_start":"94'223309","ondisk_log_start":"94'223309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":4993474944,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":232016,"num_read_kb":907962,"num_write":230949,"num_write_kb":10690755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":271,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:27:46.870746+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:27:46.870746+0000","last_peered":"2023-10-03T21:27:46.870746+0000","last_clean":"2023-10-03T21:27:46.870746+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:27:46.870746+0000","last_undegraded":"2023-10-03T21:27:46.870746+0000","last_fullsized":"2023-10-03T21:27:46.870746+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610879793,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":82321,"num_read_kb":227358,"num_write":63563,"num_write_kb":208675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63124,"num_flush":16539,"num_flush_kb":66615997,"num_evict":10788,"num_evict_kb":43530274,"num_promote":10826,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1226215424,"data_stored":1225933410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40080,"ondisk_log_size":40080,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40390741136,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1851738,"num_read_kb":7237356,"num_write":1826056,"num_write_kb":73707407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43463270400,"data_stored":48914221344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80291,"ondisk_log_size":80291,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739272,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9905072,"kb_used_data":8491816,"kb_used_omap":376,"kb_used_meta":1412871,"kb_avail":84466768,"statfs":{"total":96636764160,"available":86493970432,"internally_reserved":0,"allocated":8695619584,"data_stored":9823336743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1446780829},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.89600000000000002,"15min":0.88500000000000001},"min":{"1min":0.185,"5min":0.16800000000000001,"15min":0.16700000000000001},"max":{"1min":3.7130000000000001,"5min":7.4219999999999997,"15min":8.2720000000000002},"last":0.216},{"interface":"front","average":{"1min":0.878,"5min":1.0049999999999999,"15min":0.98399999999999999},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.9640000000000004,"5min":7.2220000000000004,"15min":15.244},"last":0.23499999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:28:27 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.456,"15min":1.1140000000000001},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":8.9299999999999997,"5min":9.0120000000000005,"15min":30.677},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2689999999999999,"15min":1.353},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":14.381,"5min":14.381,"15min":30.045999999999999},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739262,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23209420,"kb_used_data":21529716,"kb_used_omap":963,"kb_used_meta":1678716,"kb_avail":71162420,"statfs":{"total":96636764160,"available":72870318080,"internally_reserved":0,"allocated":22046429184,"data_stored":24771478701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1719005590},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,6,8,12,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:28:42 2023","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.8480000000000001,"15min":2.0390000000000001},"min":{"1min":0.20200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":10.156000000000001,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.095,"5min":1.2509999999999999,"15min":1.151},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.183},"max":{"1min":6.5819999999999999,"5min":7.3799999999999999,"15min":28.699999999999999},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:28:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.4239999999999999,"15min":1.3380000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":6.625,"5min":12.750999999999999,"15min":28.585999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.3480000000000001,"15min":1.339},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":6.1079999999999997,"5min":11.831,"15min":14.201000000000001},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359739275,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15214468,"kb_used_data":13631292,"kb_used_omap":597,"kb_used_meta":1582570,"kb_avail":79157372,"statfs":{"total":96636764160,"available":81057148928,"internally_reserved":0,"allocated":13958443008,"data_stored":15555236632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1620552609},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,6,16,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:27:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.056,"15min":1.196},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.16200000000000001},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":21.391999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4450000000000001,"15min":1.222},"min":{"1min":0.254,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":11.349,"5min":23.422000000000001,"15min":23.422000000000001},"last":5.0220000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":1.0309999999999999,"15min":1.0740000000000001},"min":{"1min":0.23400000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":0.92700000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.996,"15min":1.012},"min":{"1min":0.25600000000000001,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":1.181,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13651628032,"data_stored":15248893864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21731840000,"data_stored":24457280656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8079802368,"data_stored":9208046824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305569792,"data_stored":305499714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305442816,"data_stored":305371887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":615202816,"data_stored":615061809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:28:47.138 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:47.139 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:28:47.139 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:28:48.065 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:48.056+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:48.093 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:48.088+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:28:48.096 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:48.088+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:28:48.097 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:48.088+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:28:48.097 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:48.088+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:28:48.097 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:48.088+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:28:51.624 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:51.616+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:28:51.624 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:51.616+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:28:51.639 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:51.632+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:28:51.647 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:51.640+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:28:51.647 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:51.640+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:28:52.238 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:28:52.239 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:52.239 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:28:52.239 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:28:52.239 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:28:52.239 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:28:52.239 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:52.224+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:28:53.802 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:53.796+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:28:53.802 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:28:53.796+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:28:55.154 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.148+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:28:55.165 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.160+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:55.312 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.304+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:28:55.313 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.304+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:28:55.313 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.304+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:28:55.313 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.304+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:28:55.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:28:55.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:28:55.325 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:28:55.320+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:01.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:01.376+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:29:01.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:01.376+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:29:01.383 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:01.376+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:29:01.400 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:01.396+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:29:01.400 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:01.396+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:29:01.412 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:01.404+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:29:01.744 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:01.736+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:29:01.758 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:01.752+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:29:01.769 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:01.764+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:29:01.769 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:01.764+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:29:01.769 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:01.764+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:29:01.769 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:01.764+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:04.950 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:04.944+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:29:04.967 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:04.960+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:29:04.967 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:04.960+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:07.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:29:07.736 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:29:07.737 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:29:07.834 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2369,"stamp":"2023-10-03T21:29:06.912692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41056266282,"num_objects":14920,"num_object_clones":9472,"num_object_copies":29840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1942178,"num_read_kb":7496489,"num_write":1897638,"num_write_kb":74239189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63197,"num_flush":16610,"num_flush_kb":66906185,"num_evict":10841,"num_evict_kb":43746830,"num_promote":10877,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120930,"ondisk_log_size":120930,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48554176,"kb_used_data":43748640,"kb_used_omap":1936,"kb_used_meta":4803567,"kb_avail":234561344,"statfs":{"total":289910292480,"available":240190816256,"internally_reserved":0,"allocated":44798607360,"data_stored":50201682302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4918852820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,16,18,17,29,33,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29417501,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4137,"num_read_kb":16372,"num_write":3990,"num_write_kb":146503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":29,"num_flush":32,"num_flush_kb":130768,"num_evict":24,"num_evict_kb":98064,"num_promote":24,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.647062"},"pg_stats":[{"pgid":"2.7","version":"97'224509","reported_seq":449940,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:29:05.215152+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:29:05.215152+0000","last_peered":"2023-10-03T21:29:05.215152+0000","last_clean":"2023-10-03T21:29:05.215152+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:29:05.215152+0000","last_undegraded":"2023-10-03T21:29:05.215152+0000","last_fullsized":"2023-10-03T21:29:05.215152+0000","mapping_epoch":12,"log_start":"94'214413","ondisk_log_start":"94'214413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":668,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5053366606,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":225047,"num_read_kb":880154,"num_write":222882,"num_write_kb":10700054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'225668","reported_seq":453759,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:05.201648+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:29:05.201648+0000","last_peered":"2023-10-03T21:29:05.201648+0000","last_clean":"2023-10-03T21:29:05.201648+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:29:05.201648+0000","last_undegraded":"2023-10-03T21:29:05.201648+0000","last_fullsized":"2023-10-03T21:29:05.201648+0000","mapping_epoch":12,"log_start":"95'215614","ondisk_log_start":"95'215614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5104218431,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":227960,"num_read_kb":887828,"num_write":223211,"num_write_kb":10104116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"97'244212","reported_seq":490210,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:29:05.221748+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:29:05.221748+0000","last_peered":"2023-10-03T21:29:05.221748+0000","last_clean":"2023-10-03T21:29:05.221748+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:29:05.221748+0000","last_undegraded":"2023-10-03T21:29:05.221748+0000","last_fullsized":"2023-10-03T21:29:05.221748+0000","mapping_epoch":12,"log_start":"95'234209","ondisk_log_start":"95'234209","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":473,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 59s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5315846647,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":245639,"num_read_kb":961777,"num_write":242635,"num_write_kb":9659553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'233699","reported_seq":469574,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:05.213375+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:29:05.213375+0000","last_peered":"2023-10-03T21:29:05.213375+0000","last_clean":"2023-10-03T21:29:05.213375+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:29:05.213375+0000","last_undegraded":"2023-10-03T21:29:05.213375+0000","last_fullsized":"2023-10-03T21:29:05.213375+0000","mapping_epoch":12,"log_start":"94'223614","ondisk_log_start":"94'223614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5076029923,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":235333,"num_read_kb":918802,"num_write":231169,"num_write_kb":8858009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'35914","reported_seq":59068,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:02.997417+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:29:02.997417+0000","last_peered":"2023-10-03T21:29:02.997417+0000","last_clean":"2023-10-03T21:29:02.997417+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:29:02.997417+0000","last_undegraded":"2023-10-03T21:29:02.997417+0000","last_fullsized":"2023-10-03T21:29:02.997417+0000","mapping_epoch":15,"log_start":"84'25828","ondisk_log_start":"84'25828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":150760760,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19439,"num_read_kb":57443,"num_write":16615,"num_write_kb":55246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15606,"num_flush":4868,"num_flush_kb":19652400,"num_evict":2661,"num_evict_kb":10758989,"num_promote":2667,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'224897","reported_seq":450883,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:03.217779+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:29:03.217779+0000","last_peered":"2023-10-03T21:29:03.217779+0000","last_clean":"2023-10-03T21:29:03.217779+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:29:03.217779+0000","last_undegraded":"2023-10-03T21:29:03.217779+0000","last_fullsized":"2023-10-03T21:29:03.217779+0000","mapping_epoch":12,"log_start":"94'214811","ondisk_log_start":"94'214811","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4803912001,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":225492,"num_read_kb":880801,"num_write":222528,"num_write_kb":7733984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'35795","reported_seq":58266,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:04.954514+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:29:04.954514+0000","last_peered":"2023-10-03T21:29:04.954514+0000","last_clean":"2023-10-03T21:29:04.954514+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:29:04.954514+0000","last_undegraded":"2023-10-03T21:29:04.954514+0000","last_fullsized":"2023-10-03T21:29:04.954514+0000","mapping_epoch":15,"log_start":"84'25734","ondisk_log_start":"84'25734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146415180,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":20298,"num_read_kb":55074,"num_write":15538,"num_write_kb":50397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15747,"num_flush":3672,"num_flush_kb":14762756,"num_evict":2777,"num_evict_kb":11178496,"num_promote":2788,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'236515","reported_seq":474521,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:04.602233+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:29:04.602233+0000","last_peered":"2023-10-03T21:29:04.602233+0000","last_clean":"2023-10-03T21:29:04.602233+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:29:04.602233+0000","last_undegraded":"2023-10-03T21:29:04.602233+0000","last_fullsized":"2023-10-03T21:29:04.602233+0000","mapping_epoch":12,"log_start":"95'226415","ondisk_log_start":"95'226415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5189706149,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":237505,"num_read_kb":929326,"num_write":233949,"num_write_kb":8566439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'31517","reported_seq":52352,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:03.192721+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:29:03.192721+0000","last_peered":"2023-10-03T21:29:03.192721+0000","last_clean":"2023-10-03T21:29:03.192721+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:29:03.192721+0000","last_undegraded":"2023-10-03T21:29:03.192721+0000","last_fullsized":"2023-10-03T21:29:03.192721+0000","mapping_epoch":15,"log_start":"84'21436","ondisk_log_start":"84'21436","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146432314,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":17390,"num_read_kb":55028,"num_write":15850,"num_write_kb":52151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16447,"num_flush":4056,"num_flush_kb":16373800,"num_evict":2723,"num_evict_kb":11027753,"num_promote":2732,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"97'227773","reported_seq":457972,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:04.563876+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:29:04.563876+0000","last_peered":"2023-10-03T21:29:04.563876+0000","last_clean":"2023-10-03T21:29:04.563876+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:29:04.563876+0000","last_undegraded":"2023-10-03T21:29:04.563876+0000","last_fullsized":"2023-10-03T21:29:04.563876+0000","mapping_epoch":12,"log_start":"95'217715","ondisk_log_start":"95'217715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4906267075,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":229562,"num_read_kb":897658,"num_write":225372,"num_write_kb":7686493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'44411","reported_seq":71805,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:05.115245+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:29:05.115245+0000","last_peered":"2023-10-03T21:29:05.115245+0000","last_clean":"2023-10-03T21:29:05.115245+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:29:05.115245+0000","last_undegraded":"2023-10-03T21:29:05.115245+0000","last_fullsized":"2023-10-03T21:29:05.115245+0000","mapping_epoch":15,"log_start":"84'34331","ondisk_log_start":"84'34331","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":158952636,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25402,"num_read_kb":60645,"num_write":15852,"num_write_kb":51830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15397,"num_flush":4014,"num_flush_kb":16117229,"num_evict":2680,"num_evict_kb":10781592,"num_promote":2690,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'234350","reported_seq":467894,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:29:05.193376+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:29:05.193376+0000","last_peered":"2023-10-03T21:29:05.193376+0000","last_clean":"2023-10-03T21:29:05.193376+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:29:05.193376+0000","last_undegraded":"2023-10-03T21:29:05.193376+0000","last_fullsized":"2023-10-03T21:29:05.193376+0000","mapping_epoch":12,"log_start":"95'224309","ondisk_log_start":"95'224309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5003768192,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":233005,"num_read_kb":911862,"num_write":231900,"num_write_kb":10719023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":271,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:27:46.870746+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:27:46.870746+0000","last_peered":"2023-10-03T21:27:46.870746+0000","last_clean":"2023-10-03T21:27:46.870746+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:27:46.870746+0000","last_undegraded":"2023-10-03T21:27:46.870746+0000","last_fullsized":"2023-10-03T21:27:46.870746+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602560890,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":82529,"num_read_kb":228190,"num_write":63855,"num_write_kb":209624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63197,"num_flush":16610,"num_flush_kb":66906185,"num_evict":10841,"num_evict_kb":43746830,"num_promote":10877,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1201225728,"data_stored":1200943848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40308,"ondisk_log_size":40308,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40453115024,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1859543,"num_read_kb":7268208,"num_write":1833646,"num_write_kb":74027671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43585970176,"data_stored":48990488864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80523,"ondisk_log_size":80523,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739276,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9958720,"kb_used_data":8506296,"kb_used_omap":376,"kb_used_meta":1452039,"kb_avail":84413120,"statfs":{"total":96636764160,"available":86439034880,"internally_reserved":0,"allocated":8710447104,"data_stored":9826707824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1486888861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,5,3,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.89600000000000002,"15min":0.88500000000000001},"min":{"1min":0.185,"5min":0.16800000000000001,"15min":0.16700000000000001},"max":{"1min":3.7130000000000001,"5min":7.4219999999999997,"15min":8.2720000000000002},"last":1.855},{"interface":"front","average":{"1min":0.878,"5min":1.0049999999999999,"15min":0.98399999999999999},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.9640000000000004,"5min":7.2220000000000004,"15min":15.244},"last":1.282}]},{"osd":1,"last update":"Tue Oct 3 21:28:27 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.456,"15min":1.1140000000000001},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":8.9299999999999997,"5min":9.0120000000000005,"15min":30.677},"last":2.476},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2689999999999999,"15min":1.353},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":14.381,"5min":14.381,"15min":30.045999999999999},"last":0.42699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739266,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23331420,"kb_used_data":21583692,"kb_used_omap":963,"kb_used_meta":1746748,"kb_avail":71040420,"statfs":{"total":96636764160,"available":72745390080,"internally_reserved":0,"allocated":22101700608,"data_stored":24803206342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1788670358},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,8,7,13,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:28:42 2023","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.8480000000000001,"15min":2.0390000000000001},"min":{"1min":0.20200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":10.156000000000001,"5min":11.055999999999999,"15min":28.088999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.095,"5min":1.2509999999999999,"15min":1.151},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.183},"max":{"1min":6.5819999999999999,"5min":7.3799999999999999,"15min":28.699999999999999},"last":9.3439999999999994}]},{"osd":2,"last update":"Tue Oct 3 21:28:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.4239999999999999,"15min":1.3380000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":6.625,"5min":12.750999999999999,"15min":28.585999999999999},"last":3.5049999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.3480000000000001,"15min":1.339},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":6.1079999999999997,"5min":11.831,"15min":14.201000000000001},"last":0.23400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15264036,"kb_used_data":13658652,"kb_used_omap":597,"kb_used_meta":1604778,"kb_avail":79107804,"statfs":{"total":96636764160,"available":81006391296,"internally_reserved":0,"allocated":13986459648,"data_stored":15571768136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1643293601},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,7,13,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.0640000000000001,"15min":1.101},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.16200000000000001},"max":{"1min":4.0830000000000002,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":2.1779999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.4339999999999999,"15min":1.177},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":5.0220000000000002,"5min":23.422000000000001,"15min":23.422000000000001},"last":2.036}]},{"osd":2,"last update":"Tue Oct 3 21:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.97399999999999998,"15min":1.0429999999999999},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":2.3170000000000002,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":2.2389999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.0569999999999999,"15min":1.012},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":6.633,"5min":8.9160000000000004,"15min":11.329000000000001},"last":2.2130000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13679726592,"data_stored":15265478568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21794922496,"data_stored":24496917648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8111321088,"data_stored":9228092648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297263104,"data_stored":297193074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305438720,"data_stored":305367816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":598523904,"data_stored":598382958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:29:07.836 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:29:07.836 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:29:07.837 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:29:08.694 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:08.688+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:29:08.694 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:08.688+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:29:08.728 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:08.720+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:11.868 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:11.860+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:29:11.869 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:11.860+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:29:11.869 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:11.860+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:29:11.884 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:11.876+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:29:11.884 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:11.876+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:29:11.884 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:11.876+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:13.831 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:13.824+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:27.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:29:28.505 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:29:28.506 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:29:28.540 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2380,"stamp":"2023-10-03T21:29:28.271356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41108617240,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14793,"num_whiteouts":11,"num_read":1950022,"num_read_kb":7527517,"num_write":1905532,"num_write_kb":74556219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63179,"num_flush":16676,"num_flush_kb":67175777,"num_evict":10886,"num_evict_kb":43930534,"num_promote":10924,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120681,"ondisk_log_size":120681,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48688056,"kb_used_data":43839452,"kb_used_omap":1936,"kb_used_meta":4846639,"kb_avail":234427464,"statfs":{"total":289910292480,"available":240053723136,"internally_reserved":0,"allocated":44891598848,"data_stored":50278653127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":4962958548},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,7,10,12,24,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9400303,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3984,"num_read_kb":15800,"num_write":4017,"num_write_kb":162882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-17,"num_flush":32,"num_flush_kb":130744,"num_evict":27,"num_evict_kb":110164,"num_promote":25,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":-1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002244"},"pg_stats":[{"pgid":"2.7","version":"98'225635","reported_seq":452038,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:29:26.064285+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:29:26.064285+0000","last_peered":"2023-10-03T21:29:26.064285+0000","last_clean":"2023-10-03T21:29:26.064285+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:29:26.064285+0000","last_undegraded":"2023-10-03T21:29:26.064285+0000","last_fullsized":"2023-10-03T21:29:26.064285+0000","mapping_epoch":12,"log_start":"95'215614","ondisk_log_start":"95'215614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":897,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5054648654,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":226017,"num_read_kb":884014,"num_write":223848,"num_write_kb":10740682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'226609","reported_seq":455600,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:25.878310+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:29:25.878310+0000","last_peered":"2023-10-03T21:29:25.878310+0000","last_clean":"2023-10-03T21:29:25.878310+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:29:25.878310+0000","last_undegraded":"2023-10-03T21:29:25.878310+0000","last_fullsized":"2023-10-03T21:29:25.878310+0000","mapping_epoch":12,"log_start":"95'216514","ondisk_log_start":"95'216514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5111648575,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":228851,"num_read_kb":891328,"num_write":224152,"num_write_kb":10148772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'245387","reported_seq":492442,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:29:25.865350+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:29:25.865350+0000","last_peered":"2023-10-03T21:29:25.865350+0000","last_clean":"2023-10-03T21:29:25.865350+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:29:25.865350+0000","last_undegraded":"2023-10-03T21:29:25.865350+0000","last_fullsized":"2023-10-03T21:29:25.865350+0000","mapping_epoch":12,"log_start":"95'235309","ondisk_log_start":"95'235309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":700,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 79s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5317284343,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":246674,"num_read_kb":965861,"num_write":243656,"num_write_kb":9712601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'234675","reported_seq":471576,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:25.877552+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:29:25.877552+0000","last_peered":"2023-10-03T21:29:25.877552+0000","last_clean":"2023-10-03T21:29:25.877552+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:29:25.877552+0000","last_undegraded":"2023-10-03T21:29:25.877552+0000","last_fullsized":"2023-10-03T21:29:25.877552+0000","mapping_epoch":12,"log_start":"95'224614","ondisk_log_start":"95'224614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5083382243,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":236348,"num_read_kb":922818,"num_write":232145,"num_write_kb":8894541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'36050","reported_seq":59281,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:22.897891+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:29:22.897891+0000","last_peered":"2023-10-03T21:29:22.897891+0000","last_clean":"2023-10-03T21:29:22.897891+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:29:22.897891+0000","last_undegraded":"2023-10-03T21:29:22.897891+0000","last_fullsized":"2023-10-03T21:29:22.897891+0000","mapping_epoch":15,"log_start":"84'26028","ondisk_log_start":"84'26028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154909996,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19483,"num_read_kb":57619,"num_write":16671,"num_write_kb":55428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15594,"num_flush":4885,"num_flush_kb":19721904,"num_evict":2671,"num_evict_kb":10799857,"num_promote":2678,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'225797","reported_seq":452687,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:23.361559+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:29:23.361559+0000","last_peered":"2023-10-03T21:29:23.361559+0000","last_clean":"2023-10-03T21:29:23.361559+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:29:23.361559+0000","last_undegraded":"2023-10-03T21:29:23.361559+0000","last_fullsized":"2023-10-03T21:29:23.361559+0000","mapping_epoch":12,"log_start":"95'215711","ondisk_log_start":"95'215711","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4808307009,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":226392,"num_read_kb":884373,"num_write":223428,"num_write_kb":7757904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'35942","reported_seq":58505,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:25.807565+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:29:25.807565+0000","last_peered":"2023-10-03T21:29:25.807565+0000","last_clean":"2023-10-03T21:29:25.807565+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:29:25.807565+0000","last_undegraded":"2023-10-03T21:29:25.807565+0000","last_fullsized":"2023-10-03T21:29:25.807565+0000","mapping_epoch":15,"log_start":"84'25935","ondisk_log_start":"84'25935","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154873468,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20352,"num_read_kb":55290,"num_write":15601,"num_write_kb":50607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15795,"num_flush":3689,"num_flush_kb":14832184,"num_evict":2787,"num_evict_kb":11219308,"num_promote":2800,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"98'237505","reported_seq":476484,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:24.705494+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:29:24.705494+0000","last_peered":"2023-10-03T21:29:24.705494+0000","last_clean":"2023-10-03T21:29:24.705494+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:29:24.705494+0000","last_undegraded":"2023-10-03T21:29:24.705494+0000","last_fullsized":"2023-10-03T21:29:24.705494+0000","mapping_epoch":12,"log_start":"95'227415","ondisk_log_start":"95'227415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5197025701,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":238470,"num_read_kb":933138,"num_write":234939,"num_write_kb":8598915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'31658","reported_seq":52588,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:23.306457+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:29:23.306457+0000","last_peered":"2023-10-03T21:29:23.306457+0000","last_clean":"2023-10-03T21:29:23.306457+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:29:23.306457+0000","last_undegraded":"2023-10-03T21:29:23.306457+0000","last_fullsized":"2023-10-03T21:29:23.306457+0000","mapping_epoch":15,"log_start":"84'21637","ondisk_log_start":"84'21637","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150671651,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17456,"num_read_kb":55292,"num_write":15924,"num_write_kb":52401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16424,"num_flush":4073,"num_flush_kb":16443188,"num_evict":2734,"num_evict_kb":11072689,"num_promote":2744,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"98'228624","reported_seq":459708,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:24.670565+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:29:24.670565+0000","last_peered":"2023-10-03T21:29:24.670565+0000","last_clean":"2023-10-03T21:29:24.670565+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:29:24.670565+0000","last_undegraded":"2023-10-03T21:29:24.670565+0000","last_fullsized":"2023-10-03T21:29:24.670565+0000","mapping_epoch":12,"log_start":"95'218615","ondisk_log_start":"95'218615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4913488323,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":230443,"num_read_kb":901150,"num_write":226223,"num_write_kb":7722537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'44593","reported_seq":72083,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:25.475260+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:29:25.475260+0000","last_peered":"2023-10-03T21:29:25.475260+0000","last_clean":"2023-10-03T21:29:25.475260+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:29:25.475260+0000","last_undegraded":"2023-10-03T21:29:25.475260+0000","last_fullsized":"2023-10-03T21:29:25.475260+0000","mapping_epoch":15,"log_start":"84'34531","ondisk_log_start":"84'34531","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150625437,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25452,"num_read_kb":60845,"num_write":15919,"num_write_kb":52038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15366,"num_flush":4029,"num_flush_kb":16178501,"num_evict":2694,"num_evict_kb":10838680,"num_promote":2702,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'235339","reported_seq":469868,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:25.825809+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:29:25.825809+0000","last_peered":"2023-10-03T21:29:25.825809+0000","last_clean":"2023-10-03T21:29:25.825809+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:29:25.825809+0000","last_undegraded":"2023-10-03T21:29:25.825809+0000","last_fullsized":"2023-10-03T21:29:25.825809+0000","mapping_epoch":12,"log_start":"95'225309","ondisk_log_start":"95'225309","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5011161472,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":233978,"num_read_kb":915698,"num_write":232889,"num_write_kb":10767899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":272,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.211677+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:28:44.211677+0000","last_peered":"2023-10-03T21:28:44.211677+0000","last_clean":"2023-10-03T21:28:44.211677+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:28:44.211677+0000","last_undegraded":"2023-10-03T21:28:44.211677+0000","last_fullsized":"2023-10-03T21:28:44.211677+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":611080552,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":11,"num_read":82743,"num_read_kb":229046,"num_write":64115,"num_write_kb":210474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63179,"num_flush":16676,"num_flush_kb":67175777,"num_evict":10886,"num_evict_kb":43930534,"num_promote":10924,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218228224,"data_stored":1217946341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40112,"ondisk_log_size":40112,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40496946320,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1867173,"num_read_kb":7298380,"num_write":1841280,"num_write_kb":74343851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43674628096,"data_stored":49063151904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80470,"ondisk_log_size":80470,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739280,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9991732,"kb_used_data":8525420,"kb_used_omap":376,"kb_used_meta":1465927,"kb_avail":84380108,"statfs":{"total":96636764160,"available":86405230592,"internally_reserved":0,"allocated":8730030080,"data_stored":9843996640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1501110173},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.89600000000000002,"15min":0.88500000000000001},"min":{"1min":0.185,"5min":0.16800000000000001,"15min":0.16700000000000001},"max":{"1min":3.7130000000000001,"5min":7.4219999999999997,"15min":8.2720000000000002},"last":0.312},{"interface":"front","average":{"1min":0.878,"5min":1.0049999999999999,"15min":0.98399999999999999},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.183},"max":{"1min":4.9640000000000004,"5min":7.2220000000000004,"15min":15.244},"last":0.29499999999999998}]},{"osd":1,"last update":"Tue Oct 3 21:28:27 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.456,"15min":1.1140000000000001},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":8.9299999999999997,"5min":9.0120000000000005,"15min":30.677},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2689999999999999,"15min":1.353},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":14.381,"5min":14.381,"15min":30.045999999999999},"last":0.41299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739270,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23402412,"kb_used_data":21626912,"kb_used_omap":963,"kb_used_meta":1774524,"kb_avail":70969428,"statfs":{"total":96636764160,"available":72672694272,"internally_reserved":0,"allocated":22145957888,"data_stored":24839525205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1817112982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,5,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:28:42 2023","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.8480000000000001,"15min":2.0390000000000001},"min":{"1min":0.20200000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":10.156000000000001,"5min":11.055999999999999,"15min":28.088999999999999},"last":2.589},{"interface":"front","average":{"1min":1.095,"5min":1.2509999999999999,"15min":1.151},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.183},"max":{"1min":6.5819999999999999,"5min":7.3799999999999999,"15min":28.699999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Tue Oct 3 21:28:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.4239999999999999,"15min":1.3380000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":6.625,"5min":12.750999999999999,"15min":28.585999999999999},"last":0.376},{"interface":"front","average":{"1min":1.046,"5min":1.3480000000000001,"15min":1.339},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.20899999999999999},"max":{"1min":6.1079999999999997,"5min":11.831,"15min":14.201000000000001},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15293912,"kb_used_data":13687120,"kb_used_omap":597,"kb_used_meta":1606186,"kb_avail":79077928,"statfs":{"total":96636764160,"available":80975798272,"internally_reserved":0,"allocated":14015610880,"data_stored":15595131282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1644735393},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,6,5,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.0640000000000001,"15min":1.101},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.16200000000000001},"max":{"1min":4.0830000000000002,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.4339999999999999,"15min":1.177},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":5.0220000000000002,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.373}]},{"osd":2,"last update":"Tue Oct 3 21:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.97399999999999998,"15min":1.0429999999999999},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":2.3170000000000002,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.081,"5min":1.0569999999999999,"15min":1.012},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":6.633,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13704683520,"data_stored":15284684712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21839351808,"data_stored":24533335184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8130592768,"data_stored":9245132008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305651712,"data_stored":305581647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305569792,"data_stored":305498905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":607006720,"data_stored":606865789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:29:28.542 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:29:28.542 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:29:28.542 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:29:29.198 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:29.192+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:32.870 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:32.864+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:29:32.871 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:32.864+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:29:32.873 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:32.864+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:29:32.888 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:32.884+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:29:32.889 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:32.884+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:29:32.894 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:32.888+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:29:38.963 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:38.951+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:29:38.994 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:38.987+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:29:44.788 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:44.779+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:29:44.809 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:44.803+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:29:48.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:29:49.431 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:29:49.431 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:29:49.478 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2390,"stamp":"2023-10-03T21:29:48.279091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41153218745,"num_objects":14922,"num_object_clones":9472,"num_object_copies":29844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":1957825,"num_read_kb":7558349,"num_write":1913423,"num_write_kb":74816208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63340,"num_flush":16738,"num_flush_kb":67429129,"num_evict":10937,"num_evict_kb":44139022,"num_promote":10975,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120703,"ondisk_log_size":120703,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":48978144,"kb_used_data":43937984,"kb_used_omap":1936,"kb_used_meta":5038191,"kb_avail":234137376,"statfs":{"total":289910292480,"available":239756673024,"internally_reserved":0,"allocated":44992495616,"data_stored":50348440982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":5159107796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,4,8,9,16,11,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48423075,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":5107,"num_read_kb":20216,"num_write":5154,"num_write_kb":171541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":163,"num_flush":40,"num_flush_kb":163424,"num_evict":28,"num_evict_kb":114460,"num_promote":32,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006396"},"pg_stats":[{"pgid":"2.7","version":"98'226715","reported_seq":454103,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:29:46.638236+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:29:46.638236+0000","last_peered":"2023-10-03T21:29:46.638236+0000","last_clean":"2023-10-03T21:29:46.638236+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:29:46.638236+0000","last_undegraded":"2023-10-03T21:29:46.638236+0000","last_fullsized":"2023-10-03T21:29:46.638236+0000","mapping_epoch":12,"log_start":"95'216615","ondisk_log_start":"95'216615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1102,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 116s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5061951822,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":226957,"num_read_kb":887726,"num_write":224787,"num_write_kb":10777150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'227595","reported_seq":457488,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:46.649935+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:29:46.649935+0000","last_peered":"2023-10-03T21:29:46.649935+0000","last_clean":"2023-10-03T21:29:46.649935+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:29:46.649935+0000","last_undegraded":"2023-10-03T21:29:46.649935+0000","last_fullsized":"2023-10-03T21:29:46.649935+0000","mapping_epoch":12,"log_start":"95'217514","ondisk_log_start":"95'217514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5116068159,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":229751,"num_read_kb":894884,"num_write":225138,"num_write_kb":10209848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'246513","reported_seq":494636,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:29:46.616870+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:29:46.616870+0000","last_peered":"2023-10-03T21:29:46.616870+0000","last_clean":"2023-10-03T21:29:46.616870+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:29:46.616870+0000","last_undegraded":"2023-10-03T21:29:46.616870+0000","last_fullsized":"2023-10-03T21:29:46.616870+0000","mapping_epoch":12,"log_start":"95'236509","ondisk_log_start":"95'236509","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":924,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 100s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5318787575,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":247728,"num_read_kb":970021,"num_write":244621,"num_write_kb":9736889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'235650","reported_seq":473541,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:46.620536+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:29:46.620536+0000","last_peered":"2023-10-03T21:29:46.620536+0000","last_clean":"2023-10-03T21:29:46.620536+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:29:46.620536+0000","last_undegraded":"2023-10-03T21:29:46.620536+0000","last_fullsized":"2023-10-03T21:29:46.620536+0000","mapping_epoch":12,"log_start":"95'225614","ondisk_log_start":"95'225614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5084901859,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":237336,"num_read_kb":926734,"num_write":233120,"num_write_kb":8902529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'36201","reported_seq":59513,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:43.182012+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:29:43.182012+0000","last_peered":"2023-10-03T21:29:43.182012+0000","last_clean":"2023-10-03T21:29:43.182012+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:29:43.182012+0000","last_undegraded":"2023-10-03T21:29:43.182012+0000","last_fullsized":"2023-10-03T21:29:43.182012+0000","mapping_epoch":15,"log_start":"84'26128","ondisk_log_start":"84'26128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159096162,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":19529,"num_read_kb":57803,"num_write":16747,"num_write_kb":55676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15648,"num_flush":4906,"num_flush_kb":19807744,"num_evict":2684,"num_evict_kb":10853029,"num_promote":2692,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'226732","reported_seq":454577,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:43.642896+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:29:43.642896+0000","last_peered":"2023-10-03T21:29:43.642896+0000","last_clean":"2023-10-03T21:29:43.642896+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:29:43.642896+0000","last_undegraded":"2023-10-03T21:29:43.642896+0000","last_fullsized":"2023-10-03T21:29:43.642896+0000","mapping_epoch":12,"log_start":"95'216711","ondisk_log_start":"95'216711","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4815884609,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":227337,"num_read_kb":888101,"num_write":224363,"num_write_kb":7777932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'36086","reported_seq":58730,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:46.467620+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:29:46.467620+0000","last_peered":"2023-10-03T21:29:46.467620+0000","last_clean":"2023-10-03T21:29:46.467620+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:29:46.467620+0000","last_undegraded":"2023-10-03T21:29:46.467620+0000","last_fullsized":"2023-10-03T21:29:46.467620+0000","mapping_epoch":15,"log_start":"84'26035","ondisk_log_start":"84'26035","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":146517642,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20398,"num_read_kb":55474,"num_write":15662,"num_write_kb":50783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15809,"num_flush":3701,"num_flush_kb":14881264,"num_evict":2803,"num_evict_kb":11284708,"num_promote":2814,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"98'238442","reported_seq":478359,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:44.766957+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:29:44.766957+0000","last_peered":"2023-10-03T21:29:44.766957+0000","last_clean":"2023-10-03T21:29:44.766957+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:29:44.766957+0000","last_undegraded":"2023-10-03T21:29:44.766957+0000","last_fullsized":"2023-10-03T21:29:44.766957+0000","mapping_epoch":12,"log_start":"95'228415","ondisk_log_start":"95'228415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5198406053,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":239402,"num_read_kb":936818,"num_write":235876,"num_write_kb":8610823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'31782","reported_seq":52772,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:42.844401+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:29:42.844401+0000","last_peered":"2023-10-03T21:29:42.844401+0000","last_clean":"2023-10-03T21:29:42.844401+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:29:42.844401+0000","last_undegraded":"2023-10-03T21:29:42.844401+0000","last_fullsized":"2023-10-03T21:29:42.844401+0000","mapping_epoch":15,"log_start":"84'21737","ondisk_log_start":"84'21737","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":146460999,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17496,"num_read_kb":55452,"num_write":15994,"num_write_kb":52620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16460,"num_flush":4082,"num_flush_kb":16479972,"num_evict":2749,"num_evict_kb":11133969,"num_promote":2758,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"98'229536","reported_seq":461560,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:44.728611+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:29:44.728611+0000","last_peered":"2023-10-03T21:29:44.728611+0000","last_clean":"2023-10-03T21:29:44.728611+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:29:44.728611+0000","last_undegraded":"2023-10-03T21:29:44.728611+0000","last_fullsized":"2023-10-03T21:29:44.728611+0000","mapping_epoch":12,"log_start":"95'219515","ondisk_log_start":"95'219515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4923810243,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":231371,"num_read_kb":904814,"num_write":227135,"num_write_kb":7775221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'44779","reported_seq":72403,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:46.468898+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:29:46.468898+0000","last_peered":"2023-10-03T21:29:46.468898+0000","last_clean":"2023-10-03T21:29:46.468898+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:29:46.468898+0000","last_undegraded":"2023-10-03T21:29:46.468898+0000","last_fullsized":"2023-10-03T21:29:46.468898+0000","mapping_epoch":15,"log_start":"84'34731","ondisk_log_start":"84'34731","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":158887126,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25527,"num_read_kb":61145,"num_write":15987,"num_write_kb":52280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15423,"num_flush":4049,"num_flush_kb":16260149,"num_evict":2701,"num_evict_kb":10867316,"num_promote":2711,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'236306","reported_seq":471751,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:46.598481+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:29:46.598481+0000","last_peered":"2023-10-03T21:29:46.598481+0000","last_clean":"2023-10-03T21:29:46.598481+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:29:46.598481+0000","last_undegraded":"2023-10-03T21:29:46.598481+0000","last_fullsized":"2023-10-03T21:29:46.598481+0000","mapping_epoch":12,"log_start":"95'226209","ondisk_log_start":"95'226209","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5021856128,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":234887,"num_read_kb":919286,"num_write":233856,"num_write_kb":10812563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":272,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.211677+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:28:44.211677+0000","last_peered":"2023-10-03T21:28:44.211677+0000","last_clean":"2023-10-03T21:28:44.211677+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:28:44.211677+0000","last_undegraded":"2023-10-03T21:28:44.211677+0000","last_fullsized":"2023-10-03T21:28:44.211677+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":610961929,"num_objects":227,"num_object_clones":0,"num_object_copies":454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":82950,"num_read_kb":229874,"num_write":64390,"num_write_kb":211359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63340,"num_flush":16738,"num_flush_kb":67429129,"num_evict":10937,"num_evict_kb":44139022,"num_promote":10975,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1218031616,"data_stored":1217749957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40217,"ondisk_log_size":40217,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40541666448,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1874769,"num_read_kb":7328384,"num_write":1848896,"num_write_kb":74602955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43767427072,"data_stored":49124800800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80387,"ondisk_log_size":80387,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739284,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10081396,"kb_used_data":8562348,"kb_used_omap":376,"kb_used_meta":1518663,"kb_avail":84290444,"statfs":{"total":96636764160,"available":86313414656,"internally_reserved":0,"allocated":8767844352,"data_stored":9873250335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1555111837},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.84599999999999997},"min":{"1min":0.19600000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.4349999999999996,"5min":7.4219999999999997,"15min":7.907},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.083,"15min":0.95099999999999996},"min":{"1min":0.214,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":9.6280000000000001,"5min":9.6280000000000001,"15min":10.976000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 3 21:29:28 2023","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":1.6379999999999999,"15min":1.2030000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":9.9719999999999995,"5min":9.9719999999999995,"15min":30.677},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.708,"5min":1.208,"15min":1.3580000000000001},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":12.177,"5min":14.381,"15min":30.045999999999999},"last":0.26000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739274,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23508500,"kb_used_data":21670540,"kb_used_omap":963,"kb_used_meta":1836988,"kb_avail":70863340,"statfs":{"total":96636764160,"available":72564060160,"internally_reserved":0,"allocated":22190632960,"data_stored":24868815755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1881076118},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,4,2,3,8,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:29:43 2023","interfaces":[{"interface":"back","average":{"1min":2.6080000000000001,"5min":2.0059999999999998,"15min":2.0230000000000001},"min":{"1min":0.19700000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":15.488,"5min":15.488,"15min":28.088999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3129999999999999,"15min":1.1599999999999999},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.183},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":28.699999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Tue Oct 3 21:29:43 2023","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.5680000000000001,"15min":1.403},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":11.345000000000001,"5min":12.750999999999999,"15min":28.585999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.5169999999999999,"15min":1.3500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":9.5570000000000004,"5min":11.831,"15min":14.201000000000001},"last":9.2970000000000006}]}]},{"osd":0,"up_from":8,"seq":34359739287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15388248,"kb_used_data":13705096,"kb_used_omap":597,"kb_used_meta":1682538,"kb_avail":78983592,"statfs":{"total":96636764160,"available":80879198208,"internally_reserved":0,"allocated":14034018304,"data_stored":15606374892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1722919841},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,3,8,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.0640000000000001,"15min":1.101},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.16200000000000001},"max":{"1min":4.0830000000000002,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":2.1019999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.4339999999999999,"15min":1.177},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":5.0220000000000002,"5min":23.422000000000001,"15min":23.422000000000001},"last":0.185}]},{"osd":2,"last update":"Tue Oct 3 21:28:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.97399999999999998,"15min":1.0429999999999999},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":2.3170000000000002,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.216},{"interface":"front","average":{"1min":1.081,"5min":1.0569999999999999,"15min":1.012},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.189},"max":{"1min":6.633,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13727080448,"data_stored":15299938216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21884440576,"data_stored":24562977936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8155906048,"data_stored":9261884648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305623040,"data_stored":305553065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305475584,"data_stored":305404768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606932992,"data_stored":606792124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:29:49.480 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:29:49.480 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:29:49.480 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:29:49.923 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:49.915+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:29:49.923 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:49.915+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:29:49.924 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:49.915+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:29:49.931 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:49.923+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:29:49.939 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:49.931+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:29:49.952 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:29:49.943+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:29:56.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:56.631+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:29:56.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:56.631+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:29:56.640 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:56.631+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:29:56.650 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:56.643+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:29:56.659 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:56.647+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:29:56.659 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:29:56.647+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:00.548 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:00.535+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:30:00.549 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:00.535+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:30:00.564 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:00.559+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:00.565 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:00.559+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:30:00.565 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:00.559+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:04.489 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:04.483+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:04.489 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:04.483+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:09.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:30:10.253 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:30:10.255 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:30:10.290 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2400,"stamp":"2023-10-03T21:30:08.290887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41189873933,"num_objects":14920,"num_object_clones":9472,"num_object_copies":29840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14792,"num_whiteouts":11,"num_read":1965556,"num_read_kb":7588937,"num_write":1921308,"num_write_kb":75117060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63424,"num_flush":16804,"num_flush_kb":67698945,"num_evict":10986,"num_evict_kb":44339202,"num_promote":11022,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120663,"ondisk_log_size":120663,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49106920,"kb_used_data":44020940,"kb_used_omap":1936,"kb_used_meta":5084015,"kb_avail":234008600,"statfs":{"total":289910292480,"available":239624806400,"internally_reserved":0,"allocated":45077442560,"data_stored":50417176169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":5206031572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,4,9,18,19,18,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15040586,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":5020,"num_read_kb":19880,"num_write":5183,"num_write_kb":220694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":74,"num_flush":49,"num_flush_kb":200292,"num_evict":32,"num_evict_kb":130720,"num_promote":28,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004133"},"pg_stats":[{"pgid":"2.7","version":"98'227714","reported_seq":456037,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:30:07.293700+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:30:07.293700+0000","last_peered":"2023-10-03T21:30:07.293700+0000","last_clean":"2023-10-03T21:30:07.293700+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:30:07.293700+0000","last_undegraded":"2023-10-03T21:30:07.293700+0000","last_fullsized":"2023-10-03T21:30:07.293700+0000","mapping_epoch":12,"log_start":"96'217615","ondisk_log_start":"96'217615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1304,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 137s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5069287758,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":227869,"num_read_kb":891326,"num_write":225648,"num_write_kb":10825538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'228559","reported_seq":459371,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:07.286396+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:30:07.286396+0000","last_peered":"2023-10-03T21:30:07.286396+0000","last_clean":"2023-10-03T21:30:07.286396+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:30:07.286396+0000","last_undegraded":"2023-10-03T21:30:07.286396+0000","last_fullsized":"2023-10-03T21:30:07.286396+0000","mapping_epoch":12,"log_start":"96'218514","ondisk_log_start":"96'218514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5120500031,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":230662,"num_read_kb":898496,"num_write":226102,"num_write_kb":10283116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"98'247664","reported_seq":496813,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:30:07.310771+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:30:07.310771+0000","last_peered":"2023-10-03T21:30:07.310771+0000","last_clean":"2023-10-03T21:30:07.310771+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:30:07.310771+0000","last_undegraded":"2023-10-03T21:30:07.310771+0000","last_fullsized":"2023-10-03T21:30:07.310771+0000","mapping_epoch":12,"log_start":"96'237610","ondisk_log_start":"96'237610","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1094,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5329146359,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":248721,"num_read_kb":973937,"num_write":245654,"num_write_kb":9794097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'236624","reported_seq":475502,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:07.311034+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:30:07.311034+0000","last_peered":"2023-10-03T21:30:07.311034+0000","last_clean":"2023-10-03T21:30:07.311034+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:30:07.311034+0000","last_undegraded":"2023-10-03T21:30:07.311034+0000","last_fullsized":"2023-10-03T21:30:07.311034+0000","mapping_epoch":12,"log_start":"95'226614","ondisk_log_start":"95'226614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5089337827,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":238314,"num_read_kb":930590,"num_write":234094,"num_write_kb":8918697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'36335","reported_seq":59739,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:03.251637+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:30:03.251637+0000","last_peered":"2023-10-03T21:30:03.251637+0000","last_clean":"2023-10-03T21:30:03.251637+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:30:03.251637+0000","last_undegraded":"2023-10-03T21:30:03.251637+0000","last_fullsized":"2023-10-03T21:30:03.251637+0000","mapping_epoch":15,"log_start":"84'26328","ondisk_log_start":"84'26328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":146591085,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":19590,"num_read_kb":58043,"num_write":16818,"num_write_kb":55906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15659,"num_flush":4922,"num_flush_kb":19873184,"num_evict":2697,"num_evict_kb":10906185,"num_promote":2702,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'227629","reported_seq":456393,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:03.890906+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:30:03.890906+0000","last_peered":"2023-10-03T21:30:03.890906+0000","last_clean":"2023-10-03T21:30:03.890906+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:30:03.890906+0000","last_undegraded":"2023-10-03T21:30:03.890906+0000","last_fullsized":"2023-10-03T21:30:03.890906+0000","mapping_epoch":12,"log_start":"95'217611","ondisk_log_start":"95'217611","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4817461569,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":228250,"num_read_kb":891709,"num_write":225260,"num_write_kb":7789668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'36220","reported_seq":58966,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:07.166196+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:30:07.166196+0000","last_peered":"2023-10-03T21:30:07.166196+0000","last_clean":"2023-10-03T21:30:07.166196+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:30:07.166196+0000","last_undegraded":"2023-10-03T21:30:07.166196+0000","last_fullsized":"2023-10-03T21:30:07.166196+0000","mapping_epoch":15,"log_start":"84'26135","ondisk_log_start":"84'26135","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154889889,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20468,"num_read_kb":55754,"num_write":15716,"num_write_kb":50961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15832,"num_flush":3714,"num_flush_kb":14934428,"num_evict":2812,"num_evict_kb":11321476,"num_promote":2825,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"98'239468","reported_seq":480351,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:05.046346+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:30:05.046346+0000","last_peered":"2023-10-03T21:30:05.046346+0000","last_clean":"2023-10-03T21:30:05.046346+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:30:05.046346+0000","last_undegraded":"2023-10-03T21:30:05.046346+0000","last_fullsized":"2023-10-03T21:30:05.046346+0000","mapping_epoch":12,"log_start":"95'229415","ondisk_log_start":"95'229415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5208822181,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":240364,"num_read_kb":940634,"num_write":236902,"num_write_kb":8643495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'31902","reported_seq":52973,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:03.474100+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:30:03.474100+0000","last_peered":"2023-10-03T21:30:03.474100+0000","last_clean":"2023-10-03T21:30:03.474100+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:30:03.474100+0000","last_undegraded":"2023-10-03T21:30:03.474100+0000","last_fullsized":"2023-10-03T21:30:03.474100+0000","mapping_epoch":15,"log_start":"84'21837","ondisk_log_start":"84'21837","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150643025,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":3,"num_read":17548,"num_read_kb":55656,"num_write":16057,"num_write_kb":52822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16470,"num_flush":4100,"num_flush_kb":16553528,"num_evict":2761,"num_evict_kb":11183025,"num_promote":2771,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"98'230473","reported_seq":463383,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:05.042063+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:30:05.042063+0000","last_peered":"2023-10-03T21:30:05.042063+0000","last_clean":"2023-10-03T21:30:05.042063+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:30:05.042063+0000","last_undegraded":"2023-10-03T21:30:05.042063+0000","last_fullsized":"2023-10-03T21:30:05.042063+0000","mapping_epoch":12,"log_start":"95'220415","ondisk_log_start":"95'220415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4928410051,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":232254,"num_read_kb":908314,"num_write":228072,"num_write_kb":7811633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'44975","reported_seq":72702,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:07.202469+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:30:07.202469+0000","last_peered":"2023-10-03T21:30:07.202469+0000","last_clean":"2023-10-03T21:30:07.202469+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:30:07.202469+0000","last_undegraded":"2023-10-03T21:30:07.202469+0000","last_fullsized":"2023-10-03T21:30:07.202469+0000","mapping_epoch":15,"log_start":"84'34932","ondisk_log_start":"84'34932","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150691070,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25581,"num_read_kb":61361,"num_write":16062,"num_write_kb":52518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15463,"num_flush":4068,"num_flush_kb":16337805,"num_evict":2716,"num_evict_kb":10928516,"num_promote":2724,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'237236","reported_seq":473626,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:30:07.301915+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:30:07.301915+0000","last_peered":"2023-10-03T21:30:07.301915+0000","last_clean":"2023-10-03T21:30:07.301915+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:30:07.301915+0000","last_undegraded":"2023-10-03T21:30:07.301915+0000","last_fullsized":"2023-10-03T21:30:07.301915+0000","mapping_epoch":12,"log_start":"96'227209","ondisk_log_start":"96'227209","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5023502720,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":235829,"num_read_kb":923026,"num_write":234786,"num_write_kb":10836715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":272,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T21:28:44.211677+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:28:44.211677+0000","last_peered":"2023-10-03T21:28:44.211677+0000","last_clean":"2023-10-03T21:28:44.211677+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:28:44.211677+0000","last_undegraded":"2023-10-03T21:28:44.211677+0000","last_fullsized":"2023-10-03T21:28:44.211677+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":602815069,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":11,"num_read":83187,"num_read_kb":230814,"num_write":64653,"num_write_kb":212207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63424,"num_flush":16804,"num_flush_kb":67698945,"num_evict":10986,"num_evict_kb":44339202,"num_promote":11022,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1222569984,"data_stored":1222288570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40586468496,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1882263,"num_read_kb":7358032,"num_write":1856518,"num_write_kb":74902959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43851788288,"data_stored":49193015584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80364,"ondisk_log_size":80364,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10101788,"kb_used_data":8586900,"kb_used_omap":376,"kb_used_meta":1514503,"kb_avail":84270052,"statfs":{"total":96636764160,"available":86292533248,"internally_reserved":0,"allocated":8792985600,"data_stored":9896208553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1550851997},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.84599999999999997},"min":{"1min":0.19600000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.4349999999999996,"5min":7.4219999999999997,"15min":7.907},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.083,"15min":0.95099999999999996},"min":{"1min":0.214,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":9.6280000000000001,"5min":9.6280000000000001,"15min":10.976000000000001},"last":1.347}]},{"osd":1,"last update":"Tue Oct 3 21:29:28 2023","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":1.6379999999999999,"15min":1.2030000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":9.9719999999999995,"5min":9.9719999999999995,"15min":30.677},"last":1.327},{"interface":"front","average":{"1min":1.708,"5min":1.208,"15min":1.3580000000000001},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":12.177,"5min":14.381,"15min":30.045999999999999},"last":1.397}]}]},{"osd":1,"up_from":8,"seq":34359739278,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23581108,"kb_used_data":21715372,"kb_used_omap":963,"kb_used_meta":1864764,"kb_avail":70790732,"statfs":{"total":96636764160,"available":72489709568,"internally_reserved":0,"allocated":22236540928,"data_stored":24906265546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1909518742},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,3,11,14,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:29:43 2023","interfaces":[{"interface":"back","average":{"1min":2.6080000000000001,"5min":2.0059999999999998,"15min":2.0230000000000001},"min":{"1min":0.19700000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":15.488,"5min":15.488,"15min":28.088999999999999},"last":1.127},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3129999999999999,"15min":1.1599999999999999},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.183},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":28.699999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:29:43 2023","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.5680000000000001,"15min":1.403},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":11.345000000000001,"5min":12.750999999999999,"15min":28.585999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.5169999999999999,"15min":1.3500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":9.5570000000000004,"5min":11.831,"15min":14.201000000000001},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15424024,"kb_used_data":13718668,"kb_used_omap":597,"kb_used_meta":1704746,"kb_avail":78947816,"statfs":{"total":96636764160,"available":80842563584,"internally_reserved":0,"allocated":14047916032,"data_stored":15614702070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1745660833},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,4,5,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:29:49 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.097,"15min":1.056},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":4.1079999999999997,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":1.1850000000000001,"15min":1.167},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.1429999999999998,"5min":13.823,"15min":23.422000000000001},"last":4.2140000000000004}]},{"osd":2,"last update":"Tue Oct 3 21:29:49 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95199999999999996,"15min":1.02},"min":{"1min":0.21299999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":4.548,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":4.3789999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.99199999999999999,"15min":0.97899999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":2.4239999999999999,"5min":8.9160000000000004,"15min":11.329000000000001},"last":4.4109999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13749399552,"data_stored":15316653992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21929730048,"data_stored":24599915664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8172658688,"data_stored":9276445928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297299968,"data_stored":297230014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305651712,"data_stored":305580959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":619618304,"data_stored":619477597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:30:10.291 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:30:10.292 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:30:10.292 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:30:17.483 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:30:17.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:30:17.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:30:17.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:30:17.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:17.484 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:30:17.485 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:17.475+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:30.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:30:31.255 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:30:31.256 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:30:31.342 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2412,"stamp":"2023-10-03T21:30:30.302651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41247910265,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14792,"num_whiteouts":11,"num_read":1973634,"num_read_kb":7620901,"num_write":1929905,"num_write_kb":75535061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63532,"num_flush":16896,"num_flush_kb":68074981,"num_evict":11045,"num_evict_kb":44580362,"num_promote":11082,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120678,"ondisk_log_size":120678,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49376020,"kb_used_data":44108788,"kb_used_omap":1936,"kb_used_meta":5265263,"kb_avail":233739500,"statfs":{"total":289910292480,"available":239349248000,"internally_reserved":0,"allocated":45167398912,"data_stored":50482580009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":5391629524},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,11,7,11,19,23,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-360336,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3821,"num_read_kb":15144,"num_write":3975,"num_write_kb":162822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":112,"num_flush":35,"num_flush_kb":143072,"num_evict":30,"num_evict_kb":122708,"num_promote":25,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007259"},"pg_stats":[{"pgid":"2.7","version":"99'228749","reported_seq":457941,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:30:28.017135+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:30:28.017135+0000","last_peered":"2023-10-03T21:30:28.017135+0000","last_clean":"2023-10-03T21:30:28.017135+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:30:28.017135+0000","last_undegraded":"2023-10-03T21:30:28.017135+0000","last_fullsized":"2023-10-03T21:30:28.017135+0000","mapping_epoch":12,"log_start":"96'218715","ondisk_log_start":"96'218715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1514,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 158s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5070528846,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":228720,"num_read_kb":894694,"num_write":226540,"num_write_kb":10869926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'229474","reported_seq":461175,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:27.953920+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:30:27.953920+0000","last_peered":"2023-10-03T21:30:27.953920+0000","last_clean":"2023-10-03T21:30:27.953920+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:30:27.953920+0000","last_undegraded":"2023-10-03T21:30:27.953920+0000","last_fullsized":"2023-10-03T21:30:27.953920+0000","mapping_epoch":12,"log_start":"96'219414","ondisk_log_start":"96'219414","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5124923711,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":231540,"num_read_kb":901964,"num_write":227017,"num_write_kb":10364420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'248778","reported_seq":498920,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:30:27.954037+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:30:27.954037+0000","last_peered":"2023-10-03T21:30:27.954037+0000","last_clean":"2023-10-03T21:30:27.954037+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:30:27.954037+0000","last_undegraded":"2023-10-03T21:30:27.954037+0000","last_fullsized":"2023-10-03T21:30:27.954037+0000","mapping_epoch":12,"log_start":"96'238710","ondisk_log_start":"96'238710","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1269,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5333553655,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":249694,"num_read_kb":977789,"num_write":246648,"num_write_kb":9814429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'237620","reported_seq":477451,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:27.944148+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:30:27.944148+0000","last_peered":"2023-10-03T21:30:27.944148+0000","last_clean":"2023-10-03T21:30:27.944148+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:30:27.944148+0000","last_undegraded":"2023-10-03T21:30:27.944148+0000","last_fullsized":"2023-10-03T21:30:27.944148+0000","mapping_epoch":12,"log_start":"96'227614","ondisk_log_start":"96'227614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5102580195,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":239258,"num_read_kb":934330,"num_write":235090,"num_write_kb":8967545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'36521","reported_seq":60028,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:28.752791+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:30:28.752791+0000","last_peered":"2023-10-03T21:30:28.752791+0000","last_clean":"2023-10-03T21:30:28.752791+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:30:28.752791+0000","last_undegraded":"2023-10-03T21:30:28.752791+0000","last_fullsized":"2023-10-03T21:30:28.752791+0000","mapping_epoch":15,"log_start":"84'26428","ondisk_log_start":"84'26428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":150756731,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19652,"num_read_kb":58291,"num_write":16906,"num_write_kb":56208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15673,"num_flush":4943,"num_flush_kb":19959060,"num_evict":2709,"num_evict_kb":10955209,"num_promote":2715,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'228804","reported_seq":458702,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:29.160863+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:30:29.160863+0000","last_peered":"2023-10-03T21:30:29.160863+0000","last_clean":"2023-10-03T21:30:29.160863+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:30:29.160863+0000","last_undegraded":"2023-10-03T21:30:29.160863+0000","last_fullsized":"2023-10-03T21:30:29.160863+0000","mapping_epoch":12,"log_start":"96'218711","ondisk_log_start":"96'218711","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4825256257,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":229375,"num_read_kb":896157,"num_write":226435,"num_write_kb":7851508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'36374","reported_seq":59217,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:27.799096+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:30:27.799096+0000","last_peered":"2023-10-03T21:30:27.799096+0000","last_clean":"2023-10-03T21:30:27.799096+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:30:27.799096+0000","last_undegraded":"2023-10-03T21:30:27.799096+0000","last_fullsized":"2023-10-03T21:30:27.799096+0000","mapping_epoch":15,"log_start":"84'26335","ondisk_log_start":"84'26335","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150728371,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20523,"num_read_kb":55974,"num_write":15802,"num_write_kb":51255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15850,"num_flush":3736,"num_flush_kb":15024312,"num_evict":2824,"num_evict_kb":11370532,"num_promote":2836,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'240656","reported_seq":482710,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:30.243280+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:30:30.243280+0000","last_peered":"2023-10-03T21:30:30.243280+0000","last_clean":"2023-10-03T21:30:30.243280+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:30:30.243280+0000","last_undegraded":"2023-10-03T21:30:30.243280+0000","last_fullsized":"2023-10-03T21:30:30.243280+0000","mapping_epoch":12,"log_start":"96'230615","ondisk_log_start":"96'230615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5219352997,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":241519,"num_read_kb":945186,"num_write":238090,"num_write_kb":8689027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'32072","reported_seq":53235,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:28.974919+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:30:28.974919+0000","last_peered":"2023-10-03T21:30:28.974919+0000","last_clean":"2023-10-03T21:30:28.974919+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:30:28.974919+0000","last_undegraded":"2023-10-03T21:30:28.974919+0000","last_fullsized":"2023-10-03T21:30:28.974919+0000","mapping_epoch":15,"log_start":"84'22038","ondisk_log_start":"84'22038","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154788173,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":3,"num_read":17607,"num_read_kb":55892,"num_write":16139,"num_write_kb":53069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16466,"num_flush":4118,"num_flush_kb":16627064,"num_evict":2781,"num_evict_kb":11264765,"num_promote":2792,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'231574","reported_seq":465549,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:30.238526+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:30:30.238526+0000","last_peered":"2023-10-03T21:30:30.238526+0000","last_clean":"2023-10-03T21:30:30.238526+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:30:30.238526+0000","last_undegraded":"2023-10-03T21:30:30.238526+0000","last_fullsized":"2023-10-03T21:30:30.238526+0000","mapping_epoch":12,"log_start":"96'221515","ondisk_log_start":"96'221515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4936004035,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":233315,"num_read_kb":912514,"num_write":229173,"num_write_kb":7877317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'45154","reported_seq":72993,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:27.699699+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:30:27.699699+0000","last_peered":"2023-10-03T21:30:27.699699+0000","last_clean":"2023-10-03T21:30:27.699699+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:30:27.699699+0000","last_undegraded":"2023-10-03T21:30:27.699699+0000","last_fullsized":"2023-10-03T21:30:27.699699+0000","mapping_epoch":15,"log_start":"84'35132","ondisk_log_start":"84'35132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":150678862,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":25628,"num_read_kb":61549,"num_write":16139,"num_write_kb":52764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15543,"num_flush":4099,"num_flush_kb":16464545,"num_evict":2731,"num_evict_kb":10989856,"num_promote":2739,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'238239","reported_seq":475502,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:27.886050+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:30:27.886050+0000","last_peered":"2023-10-03T21:30:27.886050+0000","last_clean":"2023-10-03T21:30:27.886050+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:30:27.886050+0000","last_undegraded":"2023-10-03T21:30:27.886050+0000","last_fullsized":"2023-10-03T21:30:27.886050+0000","mapping_epoch":12,"log_start":"96'228209","ondisk_log_start":"96'228209","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5028168064,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":236697,"num_read_kb":926470,"num_write":235789,"num_write_kb":10885699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":273,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-03T21:29:06.935722+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:29:06.935722+0000","last_peered":"2023-10-03T21:29:06.935722+0000","last_clean":"2023-10-03T21:29:06.935722+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:29:06.935722+0000","last_undegraded":"2023-10-03T21:29:06.935722+0000","last_fullsized":"2023-10-03T21:29:06.935722+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":606952137,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":11,"num_read":83410,"num_read_kb":231706,"num_write":64986,"num_write_kb":213296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63532,"num_flush":16896,"num_flush_kb":68074981,"num_evict":11045,"num_evict_kb":44580362,"num_promote":11082,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197502464,"data_stored":1197221220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40640367760,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1890118,"num_read_kb":7389104,"num_write":1864782,"num_write_kb":75319871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43967221760,"data_stored":49283832096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80391,"ondisk_log_size":80391,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739293,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10150216,"kb_used_data":8581184,"kb_used_omap":376,"kb_used_meta":1568647,"kb_avail":84221624,"statfs":{"total":96636764160,"available":86242942976,"internally_reserved":0,"allocated":8787132416,"data_stored":9887266670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1606295453},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,0,0,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.84599999999999997},"min":{"1min":0.19600000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.4349999999999996,"5min":7.4219999999999997,"15min":7.907},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.083,"15min":0.95099999999999996},"min":{"1min":0.214,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":9.6280000000000001,"5min":9.6280000000000001,"15min":10.976000000000001},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Oct 3 21:29:28 2023","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":1.6379999999999999,"15min":1.2030000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":9.9719999999999995,"5min":9.9719999999999995,"15min":30.677},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.208,"15min":1.3580000000000001},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":12.177,"5min":14.381,"15min":30.045999999999999},"last":0.61499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739282,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23709428,"kb_used_data":21763172,"kb_used_omap":963,"kb_used_meta":1945276,"kb_avail":70662412,"statfs":{"total":96636764160,"available":72358309888,"internally_reserved":0,"allocated":22285488128,"data_stored":24943452782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":1991963030},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,10,6,6,11,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:29:43 2023","interfaces":[{"interface":"back","average":{"1min":2.6080000000000001,"5min":2.0059999999999998,"15min":2.0230000000000001},"min":{"1min":0.19700000000000001,"5min":0.16200000000000001,"15min":0.153},"max":{"1min":15.488,"5min":15.488,"15min":28.088999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3129999999999999,"15min":1.1599999999999999},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.183},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":28.699999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Oct 3 21:29:43 2023","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.5680000000000001,"15min":1.403},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":11.345000000000001,"5min":12.750999999999999,"15min":28.585999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.5169999999999999,"15min":1.3500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":9.5570000000000004,"5min":11.831,"15min":14.201000000000001},"last":4.0810000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15516376,"kb_used_data":13764432,"kb_used_omap":597,"kb_used_meta":1751338,"kb_avail":78855464,"statfs":{"total":96636764160,"available":80747995136,"internally_reserved":0,"allocated":14094778368,"data_stored":15651860557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1793371041},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,8,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:29:49 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.097,"15min":1.056},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":4.1079999999999997,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.217},{"interface":"front","average":{"1min":0.73799999999999999,"5min":1.1850000000000001,"15min":1.167},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.1429999999999998,"5min":13.823,"15min":23.422000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:29:49 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95199999999999996,"15min":1.02},"min":{"1min":0.21299999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":4.548,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.99199999999999999,"15min":0.97899999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":2.4239999999999999,"5min":8.9160000000000004,"15min":11.329000000000001},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13788041216,"data_stored":15345588136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21983182848,"data_stored":24641563792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8195997696,"data_stored":9296680168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305610752,"data_stored":305540808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301477888,"data_stored":301407233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":590413824,"data_stored":590273179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:30:31.344 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:30:31.344 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:30:31.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-03T21:30:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-03T21:30:32.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T21:30:32.897 INFO:teuthology.orchestra.run.smithi167.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T21:30:32.952 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:30:34.776 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:34.766+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:30:34.781 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:34.774+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:30:34.786 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:34.778+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:30:37.248 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.238+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:30:37.256 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.246+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:30:37.257 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.246+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:30:37.257 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.246+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:30:37.315 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.306+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:37.315 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.306+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:30:37.316 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:37.306+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:40.805 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:40.798+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:40.806 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:40.798+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:42.057 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:42.046+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:30:42.057 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:42.046+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:30:42.065 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:42.058+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:30:42.231 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:42.222+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:30:42.273 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:42.266+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:30:42.273 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:42.266+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:30:44.923 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:44.910+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:44.929 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:44.922+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:30:44.929 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:44.922+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:45.739 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:45.726+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:45.745 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:45.738+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:30:51.496 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:51.490+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:30:52.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:30:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:30:53.849 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:30:53.899 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2423,"stamp":"2023-10-03T21:30:52.310278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41277372719,"num_objects":14921,"num_object_clones":9472,"num_object_copies":29842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14794,"num_whiteouts":11,"num_read":1981276,"num_read_kb":7651161,"num_write":1937825,"num_write_kb":75827833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63458,"num_flush":16963,"num_flush_kb":68348817,"num_evict":11094,"num_evict_kb":44780542,"num_promote":11131,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29032,"num_omap_keys":505,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120678,"ondisk_log_size":120678,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49544428,"kb_used_data":44178876,"kb_used_omap":1936,"kb_used_meta":5359535,"kb_avail":233571092,"statfs":{"total":289910292480,"available":239176798208,"internally_reserved":0,"allocated":45239169024,"data_stored":50546059669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":5488164052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,11,28,24,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7020516,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4252,"num_read_kb":16848,"num_write":4400,"num_write_kb":164426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-28,"num_flush":38,"num_flush_kb":155288,"num_evict":29,"num_evict_kb":118512,"num_promote":27,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":-1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004516"},"pg_stats":[{"pgid":"2.7","version":"99'229793","reported_seq":459975,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:30:48.510198+0000","last_change":"2023-10-03T21:27:49.972548+0000","last_active":"2023-10-03T21:30:48.510198+0000","last_peered":"2023-10-03T21:30:48.510198+0000","last_clean":"2023-10-03T21:30:48.510198+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:30:48.510198+0000","last_undegraded":"2023-10-03T21:30:48.510198+0000","last_fullsized":"2023-10-03T21:30:48.510198+0000","mapping_epoch":12,"log_start":"96'219715","ondisk_log_start":"96'219715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1693,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 178s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5071839566,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":229678,"num_read_kb":898470,"num_write":227460,"num_write_kb":10918490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'230422","reported_seq":463086,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:48.487548+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:30:48.487548+0000","last_peered":"2023-10-03T21:30:48.487548+0000","last_clean":"2023-10-03T21:30:48.487548+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:30:48.487548+0000","last_undegraded":"2023-10-03T21:30:48.487548+0000","last_fullsized":"2023-10-03T21:30:48.487548+0000","mapping_epoch":12,"log_start":"96'220414","ondisk_log_start":"96'220414","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5132218687,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":232495,"num_read_kb":905728,"num_write":227965,"num_write_kb":10409040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'249879","reported_seq":501017,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:30:48.496473+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:30:48.496473+0000","last_peered":"2023-10-03T21:30:48.496473+0000","last_clean":"2023-10-03T21:30:48.496473+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:30:48.496473+0000","last_undegraded":"2023-10-03T21:30:48.496473+0000","last_fullsized":"2023-10-03T21:30:48.496473+0000","mapping_epoch":12,"log_start":"96'239810","ondisk_log_start":"96'239810","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1406,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 162s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5340795383,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":250667,"num_read_kb":981661,"num_write":247651,"num_write_kb":9855137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'238569","reported_seq":479333,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:48.470685+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:30:48.470685+0000","last_peered":"2023-10-03T21:30:48.470685+0000","last_clean":"2023-10-03T21:30:48.470685+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:30:48.470685+0000","last_undegraded":"2023-10-03T21:30:48.470685+0000","last_fullsized":"2023-10-03T21:30:48.470685+0000","mapping_epoch":12,"log_start":"96'228514","ondisk_log_start":"96'228514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5104169443,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":240182,"num_read_kb":937970,"num_write":236039,"num_write_kb":8995857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'36661","reported_seq":60262,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:49.161663+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:30:49.161663+0000","last_peered":"2023-10-03T21:30:49.161663+0000","last_clean":"2023-10-03T21:30:49.161663+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:30:49.161663+0000","last_undegraded":"2023-10-03T21:30:49.161663+0000","last_fullsized":"2023-10-03T21:30:49.161663+0000","mapping_epoch":15,"log_start":"84'26628","ondisk_log_start":"84'26628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159104383,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":19713,"num_read_kb":58535,"num_write":16964,"num_write_kb":56398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15677,"num_flush":4962,"num_flush_kb":20036716,"num_evict":2719,"num_evict_kb":10996105,"num_promote":2727,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'229726","reported_seq":460459,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:49.406059+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:30:49.406059+0000","last_peered":"2023-10-03T21:30:49.406059+0000","last_clean":"2023-10-03T21:30:49.406059+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:30:49.406059+0000","last_undegraded":"2023-10-03T21:30:49.406059+0000","last_fullsized":"2023-10-03T21:30:49.406059+0000","mapping_epoch":12,"log_start":"96'219711","ondisk_log_start":"96'219711","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4826874177,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":230199,"num_read_kb":899417,"num_write":227357,"num_write_kb":7887860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'36532","reported_seq":59457,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:48.051201+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:30:48.051201+0000","last_peered":"2023-10-03T21:30:48.051201+0000","last_clean":"2023-10-03T21:30:48.051201+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:30:48.051201+0000","last_undegraded":"2023-10-03T21:30:48.051201+0000","last_fullsized":"2023-10-03T21:30:48.051201+0000","mapping_epoch":15,"log_start":"84'26435","ondisk_log_start":"84'26435","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":142356120,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20570,"num_read_kb":56162,"num_write":15873,"num_write_kb":51485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15823,"num_flush":3753,"num_flush_kb":15093844,"num_evict":2837,"num_evict_kb":11423660,"num_promote":2847,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'241664","reported_seq":484727,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:50.388915+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:30:50.388915+0000","last_peered":"2023-10-03T21:30:50.388915+0000","last_clean":"2023-10-03T21:30:50.388915+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:30:50.388915+0000","last_undegraded":"2023-10-03T21:30:50.388915+0000","last_fullsized":"2023-10-03T21:30:50.388915+0000","mapping_epoch":12,"log_start":"96'231615","ondisk_log_start":"96'231615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5223670181,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":242525,"num_read_kb":949182,"num_write":239098,"num_write_kb":8717563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'32191","reported_seq":53429,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:49.211494+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:30:49.211494+0000","last_peered":"2023-10-03T21:30:49.211494+0000","last_clean":"2023-10-03T21:30:49.211494+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:30:49.211494+0000","last_undegraded":"2023-10-03T21:30:49.211494+0000","last_fullsized":"2023-10-03T21:30:49.211494+0000","mapping_epoch":15,"log_start":"84'22138","ondisk_log_start":"84'22138","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150618399,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":3,"num_read":17658,"num_read_kb":56092,"num_write":16199,"num_write_kb":53267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16420,"num_flush":4131,"num_flush_kb":16680148,"num_evict":2791,"num_evict_kb":11305577,"num_promote":2801,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'232492","reported_seq":467340,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:50.372672+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:30:50.372672+0000","last_peered":"2023-10-03T21:30:50.372672+0000","last_clean":"2023-10-03T21:30:50.372672+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:30:50.372672+0000","last_undegraded":"2023-10-03T21:30:50.372672+0000","last_fullsized":"2023-10-03T21:30:50.372672+0000","mapping_epoch":12,"log_start":"96'222415","ondisk_log_start":"96'222415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4937404867,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":234184,"num_read_kb":915970,"num_write":230091,"num_write_kb":7913673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'45362","reported_seq":73298,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:48.040007+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:30:48.040007+0000","last_peered":"2023-10-03T21:30:48.040007+0000","last_clean":"2023-10-03T21:30:48.040007+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:30:48.040007+0000","last_undegraded":"2023-10-03T21:30:48.040007+0000","last_fullsized":"2023-10-03T21:30:48.040007+0000","mapping_epoch":15,"log_start":"84'35332","ondisk_log_start":"84'35332","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154922313,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":25674,"num_read_kb":61733,"num_write":16217,"num_write_kb":53010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15538,"num_flush":4117,"num_flush_kb":16538109,"num_evict":2747,"num_evict_kb":11055200,"num_promote":2756,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'239224","reported_seq":477419,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:48.487490+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:30:48.487490+0000","last_peered":"2023-10-03T21:30:48.487490+0000","last_clean":"2023-10-03T21:30:48.487490+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:30:48.487490+0000","last_undegraded":"2023-10-03T21:30:48.487490+0000","last_fullsized":"2023-10-03T21:30:48.487490+0000","mapping_epoch":12,"log_start":"96'229209","ondisk_log_start":"96'229209","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5032808832,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":237625,"num_read_kb":930150,"num_write":236774,"num_write_kb":10914159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":274,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:21.940372+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:30:21.940372+0000","last_peered":"2023-10-03T21:30:21.940372+0000","last_clean":"2023-10-03T21:30:21.940372+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:30:21.940372+0000","last_undegraded":"2023-10-03T21:30:21.940372+0000","last_fullsized":"2023-10-03T21:30:21.940372+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":607001215,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":11,"num_read":83615,"num_read_kb":232522,"num_write":65253,"num_write_kb":214160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63458,"num_flush":16963,"num_flush_kb":68348817,"num_evict":11094,"num_evict_kb":44780542,"num_promote":11131,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1222651904,"data_stored":1222370620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40669781136,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1897555,"num_read_kb":7418548,"num_write":1872435,"num_write_kb":75611779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15186,"num_omap_keys":288,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44026458112,"data_stored":49334757664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80366,"ondisk_log_size":80366,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739297,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10198792,"kb_used_data":8615872,"kb_used_omap":376,"kb_used_meta":1582535,"kb_avail":84173048,"statfs":{"total":96636764160,"available":86193201152,"internally_reserved":0,"allocated":8822652928,"data_stored":9921222436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1620516765},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:30:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.76200000000000001,"15min":0.87},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.399,"5min":7.4219999999999997,"15min":7.907},"last":0.224},{"interface":"front","average":{"1min":1.073,"5min":1.0169999999999999,"15min":0.98299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":4.6180000000000003,"5min":9.6280000000000001,"15min":10.976000000000001},"last":0.25800000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:30:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.3859999999999999,"15min":1.1699999999999999},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":4.1230000000000002,"5min":9.9719999999999995,"15min":30.677},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.3220000000000001,"15min":1.3879999999999999},"min":{"1min":0.26000000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":6.2009999999999996,"5min":14.381,"15min":30.045999999999999},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739286,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23758144,"kb_used_data":21779944,"kb_used_omap":963,"kb_used_meta":1977212,"kb_avail":70613696,"statfs":{"total":96636764160,"available":72308424704,"internally_reserved":0,"allocated":22302662656,"data_stored":24956478030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":2024665494},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8,12,8,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:30:43 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.819,"15min":1.732},"min":{"1min":0.20100000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.7440000000000002,"5min":15.488,"15min":17.309999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.2529999999999999,"15min":1.1699999999999999},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":13.489000000000001,"5min":13.489000000000001,"15min":28.699999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:30:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.577,"15min":1.419},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":5.0060000000000002,"5min":11.345000000000001,"15min":28.585999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.536,"15min":1.395},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":6.7089999999999996,"5min":9.5570000000000004,"15min":14.201000000000001},"last":0.38800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15587492,"kb_used_data":13783060,"kb_used_omap":597,"kb_used_meta":1799786,"kb_avail":78784348,"statfs":{"total":96636764160,"available":80675172352,"internally_reserved":0,"allocated":14113853440,"data_stored":15668359203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1842981793},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,10,12,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:29:49 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.097,"15min":1.056},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":4.1079999999999997,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.249},{"interface":"front","average":{"1min":0.73799999999999999,"5min":1.1850000000000001,"15min":1.167},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.1429999999999998,"5min":13.823,"15min":23.422000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Oct 3 21:29:49 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95199999999999996,"15min":1.02},"min":{"1min":0.21299999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":4.548,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.99199999999999999,"15min":0.97899999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":2.4239999999999999,"5min":8.9160000000000004,"15min":11.329000000000001},"last":1.071}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13807140864,"data_stored":15362078632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22012874752,"data_stored":24667106448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8206442496,"data_stored":9305572584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309788672,"data_stored":309718686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297349120,"data_stored":297278433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":615514112,"data_stored":615373501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:30:53.901 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:30:53.901 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:30:53.901 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:30:56.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:56.978+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:30:56.999 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:56.978+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:30:57.000 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:56.978+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:30:57.040 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:57.030+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:30:57.057 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:30:57.050+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:30:58.558 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:58.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:30:58.558 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:58.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:30:58.558 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:58.550+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:30:58.577 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:58.570+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:30:58.577 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:30:58.570+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:31:04.033 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.018+0000 7f27ba5d7640 -1 reset not still connected to 0x5569144e6750 2023-10-03T21:31:04.033 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.018+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c2a8f0 2023-10-03T21:31:04.033 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.018+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c5f040 2023-10-03T21:31:04.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:31:04.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:31:04.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:31:04.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:31:04.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:31:04.045 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:31:04.046 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:04.034+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:31:08.096 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.090+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:31:08.100 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.094+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:31:08.100 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.094+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:31:08.100 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.094+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:31:08.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.106+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:31:08.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.106+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:31:08.113 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:08.106+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:31:10.911 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:10.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:31:10.911 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:10.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:31:10.912 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:10.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:31:10.912 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:10.902+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:31:11.092 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.082+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:31:11.108 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.098+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:31:11.108 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.098+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:31:11.108 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.098+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:31:11.131 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:31:11.131 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:31:11.131 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:11.122+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:31:13.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:31:14.619 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:31:14.620 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:31:14.684 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2434,"stamp":"2023-10-03T21:31:14.315050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41330367059,"num_objects":14924,"num_object_clones":9472,"num_object_copies":29848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14792,"num_whiteouts":11,"num_read":1990248,"num_read_kb":7686757,"num_write":1947176,"num_write_kb":76216089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63750,"num_flush":17045,"num_flush_kb":68683833,"num_evict":11144,"num_evict_kb":44984954,"num_promote":11184,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30482,"num_omap_keys":527,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120774,"ondisk_log_size":120774,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49758672,"kb_used_data":44279600,"kb_used_omap":1936,"kb_used_meta":5477103,"kb_avail":233356848,"statfs":{"total":289910292480,"available":238957412352,"internally_reserved":0,"allocated":45342310400,"data_stored":50639006616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":5608553684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,1,3,8,32,21,24],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28168422,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5277,"num_read_kb":20928,"num_write":5408,"num_write_kb":217353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":230,"num_flush":46,"num_flush_kb":187836,"num_evict":30,"num_evict_kb":122608,"num_promote":32,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"2.7","version":"99'231073","reported_seq":462396,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:14.173773+0000","last_change":"2023-10-03T21:30:57.545790+0000","last_active":"2023-10-03T21:31:14.173773+0000","last_peered":"2023-10-03T21:31:14.173773+0000","last_clean":"2023-10-03T21:31:14.173773+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:31:14.173773+0000","last_undegraded":"2023-10-03T21:31:14.173773+0000","last_fullsized":"2023-10-03T21:31:14.173773+0000","mapping_epoch":12,"log_start":"97'221015","ondisk_log_start":"97'221015","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'230269","last_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_deep_scrub":"99'230269","last_deep_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_clean_scrub_stamp":"2023-10-03T21:30:57.545623+0000","objects_scrubbed":1786,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":188,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:27:25.972550+0000","scrub_duration":187.567402794,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5082587470,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":230798,"num_read_kb":902902,"num_write":228678,"num_write_kb":10996902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'231627","reported_seq":465469,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:14.166676+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:31:14.166676+0000","last_peered":"2023-10-03T21:31:14.166676+0000","last_clean":"2023-10-03T21:31:14.166676+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:31:14.166676+0000","last_undegraded":"2023-10-03T21:31:14.166676+0000","last_fullsized":"2023-10-03T21:31:14.166676+0000","mapping_epoch":12,"log_start":"96'221614","ondisk_log_start":"96'221614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5133910335,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":233666,"num_read_kb":910384,"num_write":229170,"num_write_kb":10479156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'251309","reported_seq":503821,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:31:14.187419+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:31:14.187419+0000","last_peered":"2023-10-03T21:31:14.187419+0000","last_clean":"2023-10-03T21:31:14.187419+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:31:14.187419+0000","last_undegraded":"2023-10-03T21:31:14.187419+0000","last_fullsized":"2023-10-03T21:31:14.187419+0000","mapping_epoch":12,"log_start":"97'241210","ondisk_log_start":"97'241210","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1592,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 188s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5342634487,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":252019,"num_read_kb":987029,"num_write":248955,"num_write_kb":9913321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'239847","reported_seq":481831,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:14.180056+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:31:14.180056+0000","last_peered":"2023-10-03T21:31:14.180056+0000","last_clean":"2023-10-03T21:31:14.180056+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:31:14.180056+0000","last_undegraded":"2023-10-03T21:31:14.180056+0000","last_fullsized":"2023-10-03T21:31:14.180056+0000","mapping_epoch":12,"log_start":"96'229814","ondisk_log_start":"96'229814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5109047779,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":241390,"num_read_kb":942742,"num_write":237317,"num_write_kb":9053965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'36793","reported_seq":60472,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:09.233789+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:31:09.233789+0000","last_peered":"2023-10-03T21:31:09.233789+0000","last_clean":"2023-10-03T21:31:09.233789+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:31:09.233789+0000","last_undegraded":"2023-10-03T21:31:09.233789+0000","last_fullsized":"2023-10-03T21:31:09.233789+0000","mapping_epoch":15,"log_start":"84'26728","ondisk_log_start":"84'26728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":159006140,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":19758,"num_read_kb":58715,"num_write":17030,"num_write_kb":56616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15738,"num_flush":4978,"num_flush_kb":20102152,"num_evict":2730,"num_evict_kb":11041109,"num_promote":2738,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'230694","reported_seq":462276,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:09.458932+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:31:09.458932+0000","last_peered":"2023-10-03T21:31:09.458932+0000","last_clean":"2023-10-03T21:31:09.458932+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:31:09.458932+0000","last_undegraded":"2023-10-03T21:31:09.458932+0000","last_fullsized":"2023-10-03T21:31:09.458932+0000","mapping_epoch":12,"log_start":"96'220611","ondisk_log_start":"96'220611","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4831441217,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":231042,"num_read_kb":902757,"num_write":228325,"num_write_kb":7920360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'36751","reported_seq":59774,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:14.105290+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:31:14.105290+0000","last_peered":"2023-10-03T21:31:14.105290+0000","last_clean":"2023-10-03T21:31:14.105290+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:31:14.105290+0000","last_undegraded":"2023-10-03T21:31:14.105290+0000","last_fullsized":"2023-10-03T21:31:14.105290+0000","mapping_epoch":15,"log_start":"84'26735","ondisk_log_start":"84'26735","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150822614,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":20629,"num_read_kb":56394,"num_write":15957,"num_write_kb":51767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15885,"num_flush":3772,"num_flush_kb":15171412,"num_evict":2850,"num_evict_kb":11476752,"num_promote":2862,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'242661","reported_seq":486659,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:10.527118+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:31:10.527118+0000","last_peered":"2023-10-03T21:31:10.527118+0000","last_clean":"2023-10-03T21:31:10.527118+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:31:10.527118+0000","last_undegraded":"2023-10-03T21:31:10.527118+0000","last_fullsized":"2023-10-03T21:31:10.527118+0000","mapping_epoch":12,"log_start":"96'232615","ondisk_log_start":"96'232615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5228102053,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":243451,"num_read_kb":952846,"num_write":240095,"num_write_kb":8758315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'32319","reported_seq":53636,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:09.175090+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:31:09.175090+0000","last_peered":"2023-10-03T21:31:09.175090+0000","last_clean":"2023-10-03T21:31:09.175090+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:31:09.175090+0000","last_undegraded":"2023-10-03T21:31:09.175090+0000","last_fullsized":"2023-10-03T21:31:09.175090+0000","mapping_epoch":15,"log_start":"84'22238","ondisk_log_start":"84'22238","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154808738,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":17704,"num_read_kb":56272,"num_write":16262,"num_write_kb":53465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16551,"num_flush":4153,"num_flush_kb":16769968,"num_evict":2804,"num_evict_kb":11358749,"num_promote":2815,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'233388","reported_seq":469148,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:10.502536+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:31:10.502536+0000","last_peered":"2023-10-03T21:31:10.502536+0000","last_clean":"2023-10-03T21:31:10.502536+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:31:10.502536+0000","last_undegraded":"2023-10-03T21:31:10.502536+0000","last_fullsized":"2023-10-03T21:31:10.502536+0000","mapping_epoch":12,"log_start":"96'223315","ondisk_log_start":"96'223315","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4941816259,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":235090,"num_read_kb":919578,"num_write":230987,"num_write_kb":7925413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'45626","reported_seq":73706,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:14.126672+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:31:14.126672+0000","last_peered":"2023-10-03T21:31:14.126672+0000","last_clean":"2023-10-03T21:31:14.126672+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:31:14.126672+0000","last_undegraded":"2023-10-03T21:31:14.126672+0000","last_fullsized":"2023-10-03T21:31:14.126672+0000","mapping_epoch":15,"log_start":"84'35532","ondisk_log_start":"84'35532","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154942831,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25761,"num_read_kb":62081,"num_write":16290,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15576,"num_flush":4142,"num_flush_kb":16640301,"num_evict":2760,"num_evict_kb":11108344,"num_promote":2769,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'240423","reported_seq":479833,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:14.165683+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:31:14.165683+0000","last_peered":"2023-10-03T21:31:14.165683+0000","last_clean":"2023-10-03T21:31:14.165683+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:31:14.165683+0000","last_undegraded":"2023-10-03T21:31:14.165683+0000","last_fullsized":"2023-10-03T21:31:14.165683+0000","mapping_epoch":12,"log_start":"96'230409","ondisk_log_start":"96'230409","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5040656768,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":238834,"num_read_kb":934966,"num_write":237973,"num_write_kb":10951667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":274,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:21.940372+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:30:21.940372+0000","last_peered":"2023-10-03T21:30:21.940372+0000","last_clean":"2023-10-03T21:30:21.940372+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:30:21.940372+0000","last_undegraded":"2023-10-03T21:30:21.940372+0000","last_fullsized":"2023-10-03T21:30:21.940372+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":619580323,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":11,"num_read":83852,"num_read_kb":233462,"num_write":65539,"num_write_kb":215096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63750,"num_flush":17045,"num_flush_kb":68683833,"num_evict":11144,"num_evict_kb":44984954,"num_promote":11184,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1231044608,"data_stored":1230763846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40256,"ondisk_log_size":40256,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40710196368,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1906290,"num_read_kb":7453204,"num_write":1881500,"num_write_kb":75999099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16636,"num_omap_keys":310,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44104060928,"data_stored":49402255648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80419,"ondisk_log_size":80419,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739301,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10229788,"kb_used_data":8632980,"kb_used_omap":376,"kb_used_meta":1596423,"kb_avail":84142052,"statfs":{"total":96636764160,"available":86161461248,"internally_reserved":0,"allocated":8840171520,"data_stored":9936889928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1634738077},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,10,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:30:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.76200000000000001,"15min":0.87},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.399,"5min":7.4219999999999997,"15min":7.907},"last":0.438},{"interface":"front","average":{"1min":1.073,"5min":1.0169999999999999,"15min":0.98299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":4.6180000000000003,"5min":9.6280000000000001,"15min":10.976000000000001},"last":0.32300000000000001}]},{"osd":1,"last update":"Tue Oct 3 21:30:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.3859999999999999,"15min":1.1699999999999999},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":4.1230000000000002,"5min":9.9719999999999995,"15min":30.677},"last":0.23300000000000001},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.3220000000000001,"15min":1.3879999999999999},"min":{"1min":0.26000000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":6.2009999999999996,"5min":14.381,"15min":30.045999999999999},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23897624,"kb_used_data":21837968,"kb_used_omap":963,"kb_used_meta":2058684,"kb_avail":70474216,"statfs":{"total":96636764160,"available":72165597184,"internally_reserved":0,"allocated":22362079232,"data_stored":25010270898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":2108092822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,14,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:30:43 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.819,"15min":1.732},"min":{"1min":0.20100000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.7440000000000002,"5min":15.488,"15min":17.309999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.2529999999999999,"15min":1.1699999999999999},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":13.489000000000001,"5min":13.489000000000001,"15min":28.699999999999999},"last":0.27000000000000002}]},{"osd":2,"last update":"Tue Oct 3 21:30:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.577,"15min":1.419},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":5.0060000000000002,"5min":11.345000000000001,"15min":28.585999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.536,"15min":1.395},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":6.7089999999999996,"5min":9.5570000000000004,"15min":14.201000000000001},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359739304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15631260,"kb_used_data":13808652,"kb_used_omap":597,"kb_used_meta":1821994,"kb_avail":78740580,"statfs":{"total":96636764160,"available":80630353920,"internally_reserved":0,"allocated":14140059648,"data_stored":15691845790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1865722785},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,2,8,3,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:30:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.88,"15min":1.014},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":4.3680000000000003,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.874},{"interface":"front","average":{"1min":1.034,"5min":1.1220000000000001,"15min":1.194},"min":{"1min":0.26400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":5.9119999999999999,"5min":13.823,"15min":23.422000000000001},"last":0.442}]},{"osd":2,"last update":"Tue Oct 3 21:30:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.77200000000000002,"15min":0.99399999999999999},"min":{"1min":0.249,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":5.9589999999999996,"5min":6.0819999999999999,"15min":8.9510000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.375,"5min":0.90900000000000003,"15min":0.96599999999999997},"min":{"1min":0.27700000000000002,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":5.9870000000000001,"5min":7.359,"15min":11.329000000000001},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13829345280,"data_stored":15381571496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22054858752,"data_stored":24703560848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8219856896,"data_stored":9317123304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313880576,"data_stored":313810782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305836032,"data_stored":305765445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":611328000,"data_stored":611187619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:31:14.686 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:31:14.687 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:31:31.547 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.541+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:31:31.555 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.549+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:31:31.555 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.549+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:31:31.555 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.549+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:31:31.566 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.561+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:31:31.567 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.561+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:31:31.567 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:31.561+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:31:32.475 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.469+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c70270 2023-10-03T21:31:32.476 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.469+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c751e0 2023-10-03T21:31:32.476 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.469+0000 7f27ba5d7640 -1 reset not still connected to 0x556914c93520 2023-10-03T21:31:32.476 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.469+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:31:32.496 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.485+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:31:32.497 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.489+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:31:32.497 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.489+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:31:32.498 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.489+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:31:32.498 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.489+0000 7f27ba5d7640 -1 reset not still connected to 0x556915f20820 2023-10-03T21:31:32.524 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:32.517+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:31:34.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:31:35.496 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:31:35.497 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:31:35.540 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2445,"stamp":"2023-10-03T21:31:34.320555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41356360180,"num_objects":14923,"num_object_clones":9472,"num_object_copies":29846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14789,"num_whiteouts":11,"num_read":1997127,"num_read_kb":7714045,"num_write":1954182,"num_write_kb":76517498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63655,"num_flush":17114,"num_flush_kb":68965865,"num_evict":11186,"num_evict_kb":45156690,"num_promote":11225,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30482,"num_omap_keys":527,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120751,"ondisk_log_size":120751,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49942376,"kb_used_data":44346768,"kb_used_omap":1936,"kb_used_meta":5593647,"kb_avail":233173144,"statfs":{"total":289910292480,"available":238769299456,"internally_reserved":0,"allocated":45411090432,"data_stored":50678700831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1983276,"internal_metadata":5727894740},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,8,16,30,23,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35856289,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":4025,"num_read_kb":15964,"num_write":4116,"num_write_kb":179665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-95,"num_flush":38,"num_flush_kb":155312,"num_evict":19,"num_evict_kb":77684,"num_promote":22,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003589"},"pg_stats":[{"pgid":"2.7","version":"99'231829","reported_seq":463898,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.444773+0000","last_change":"2023-10-03T21:30:57.545790+0000","last_active":"2023-10-03T21:31:29.444773+0000","last_peered":"2023-10-03T21:31:29.444773+0000","last_clean":"2023-10-03T21:31:29.444773+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:31:29.444773+0000","last_undegraded":"2023-10-03T21:31:29.444773+0000","last_fullsized":"2023-10-03T21:31:29.444773+0000","mapping_epoch":12,"log_start":"97'221815","ondisk_log_start":"97'221815","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'230269","last_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_deep_scrub":"99'230269","last_deep_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_clean_scrub_stamp":"2023-10-03T21:30:57.545623+0000","objects_scrubbed":1786,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":188,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:27:25.972550+0000","scrub_duration":187.567402794,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5089272142,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":231542,"num_read_kb":905854,"num_write":229434,"num_write_kb":11048898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'232406","reported_seq":466994,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.402079+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:31:29.402079+0000","last_peered":"2023-10-03T21:31:29.402079+0000","last_clean":"2023-10-03T21:31:29.402079+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:31:29.402079+0000","last_undegraded":"2023-10-03T21:31:29.402079+0000","last_fullsized":"2023-10-03T21:31:29.402079+0000","mapping_epoch":12,"log_start":"97'222314","ondisk_log_start":"97'222314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5137928511,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":234405,"num_read_kb":913296,"num_write":229949,"num_write_kb":10535328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"99'252168","reported_seq":505494,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:31:29.444493+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:31:29.444493+0000","last_peered":"2023-10-03T21:31:29.444493+0000","last_clean":"2023-10-03T21:31:29.444493+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:31:29.444493+0000","last_undegraded":"2023-10-03T21:31:29.444493+0000","last_fullsized":"2023-10-03T21:31:29.444493+0000","mapping_epoch":12,"log_start":"97'242111","ondisk_log_start":"97'242111","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1684,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 203s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5349700087,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":252819,"num_read_kb":990201,"num_write":249754,"num_write_kb":9945057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'240584","reported_seq":483271,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.402395+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:31:29.402395+0000","last_peered":"2023-10-03T21:31:29.402395+0000","last_clean":"2023-10-03T21:31:29.402395+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:31:29.402395+0000","last_undegraded":"2023-10-03T21:31:29.402395+0000","last_fullsized":"2023-10-03T21:31:29.402395+0000","mapping_epoch":12,"log_start":"97'230514","ondisk_log_start":"97'230514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5110231523,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":242094,"num_read_kb":945546,"num_write":238054,"num_write_kb":9081437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'36950","reported_seq":60738,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:28.866139+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:31:28.866139+0000","last_peered":"2023-10-03T21:31:28.866139+0000","last_clean":"2023-10-03T21:31:28.866139+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:31:28.866139+0000","last_undegraded":"2023-10-03T21:31:28.866139+0000","last_fullsized":"2023-10-03T21:31:28.866139+0000","mapping_epoch":15,"log_start":"84'26928","ondisk_log_start":"84'26928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":154815918,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":19825,"num_read_kb":58983,"num_write":17108,"num_write_kb":56878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15724,"num_flush":5005,"num_flush_kb":20212532,"num_evict":2742,"num_evict_kb":11090165,"num_promote":2749,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'231697","reported_seq":464211,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.571367+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:31:29.571367+0000","last_peered":"2023-10-03T21:31:29.571367+0000","last_clean":"2023-10-03T21:31:29.571367+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:31:29.571367+0000","last_undegraded":"2023-10-03T21:31:29.571367+0000","last_fullsized":"2023-10-03T21:31:29.571367+0000","mapping_epoch":12,"log_start":"97'221611","ondisk_log_start":"97'221611","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4833100097,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":231972,"num_read_kb":906453,"num_write":229328,"num_write_kb":7957044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'36866","reported_seq":59951,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.190329+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:31:29.190329+0000","last_peered":"2023-10-03T21:31:29.190329+0000","last_clean":"2023-10-03T21:31:29.190329+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:31:29.190329+0000","last_undegraded":"2023-10-03T21:31:29.190329+0000","last_fullsized":"2023-10-03T21:31:29.190329+0000","mapping_epoch":15,"log_start":"84'26835","ondisk_log_start":"84'26835","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154939062,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20666,"num_read_kb":56542,"num_write":16002,"num_write_kb":51917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15853,"num_flush":3783,"num_flush_kb":15216408,"num_evict":2857,"num_evict_kb":11505384,"num_promote":2870,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'243641","reported_seq":488704,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:30.638268+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:31:30.638268+0000","last_peered":"2023-10-03T21:31:30.638268+0000","last_clean":"2023-10-03T21:31:30.638268+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:31:30.638268+0000","last_undegraded":"2023-10-03T21:31:30.638268+0000","last_fullsized":"2023-10-03T21:31:30.638268+0000","mapping_epoch":12,"log_start":"97'233615","ondisk_log_start":"97'233615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5229490597,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":244511,"num_read_kb":957054,"num_write":241075,"num_write_kb":8782667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'32436","reported_seq":53828,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.399110+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:31:29.399110+0000","last_peered":"2023-10-03T21:31:29.399110+0000","last_clean":"2023-10-03T21:31:29.399110+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:31:29.399110+0000","last_undegraded":"2023-10-03T21:31:29.399110+0000","last_fullsized":"2023-10-03T21:31:29.399110+0000","mapping_epoch":15,"log_start":"84'22338","ondisk_log_start":"84'22338","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":150581644,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":17756,"num_read_kb":56480,"num_write":16325,"num_write_kb":53668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16529,"num_flush":4165,"num_flush_kb":16818988,"num_evict":2816,"num_evict_kb":11407801,"num_promote":2826,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'234366","reported_seq":471056,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:30.610662+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:31:30.610662+0000","last_peered":"2023-10-03T21:31:30.610662+0000","last_clean":"2023-10-03T21:31:30.610662+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:31:30.610662+0000","last_undegraded":"2023-10-03T21:31:30.610662+0000","last_fullsized":"2023-10-03T21:31:30.610662+0000","mapping_epoch":12,"log_start":"97'224315","ondisk_log_start":"97'224315","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4943393219,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":236015,"num_read_kb":923234,"num_write":231965,"num_write_kb":7966093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'45794","reported_seq":73952,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.372174+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:31:29.372174+0000","last_peered":"2023-10-03T21:31:29.372174+0000","last_clean":"2023-10-03T21:31:29.372174+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:31:29.372174+0000","last_undegraded":"2023-10-03T21:31:29.372174+0000","last_fullsized":"2023-10-03T21:31:29.372174+0000","mapping_epoch":15,"log_start":"84'35732","ondisk_log_start":"84'35732","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154864980,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":25803,"num_read_kb":62249,"num_write":16349,"num_write_kb":53438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15549,"num_flush":4161,"num_flush_kb":16717937,"num_evict":2771,"num_evict_kb":11153340,"num_promote":2780,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'241152","reported_seq":481346,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:31:29.401657+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:31:29.401657+0000","last_peered":"2023-10-03T21:31:29.401657+0000","last_clean":"2023-10-03T21:31:29.401657+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:31:29.401657+0000","last_undegraded":"2023-10-03T21:31:29.401657+0000","last_fullsized":"2023-10-03T21:31:29.401657+0000","mapping_epoch":12,"log_start":"97'231109","ondisk_log_start":"97'231109","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5047452032,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":239613,"num_read_kb":938062,"num_write":238702,"num_write_kb":10983179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":274,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T21:30:21.940372+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:30:21.940372+0000","last_peered":"2023-10-03T21:30:21.940372+0000","last_clean":"2023-10-03T21:30:21.940372+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:30:21.940372+0000","last_undegraded":"2023-10-03T21:30:21.940372+0000","last_fullsized":"2023-10-03T21:30:21.940372+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":615201604,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":11,"num_read":84050,"num_read_kb":234254,"num_write":65784,"num_write_kb":215901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63655,"num_flush":17114,"num_flush_kb":68965865,"num_evict":11186,"num_evict_kb":45156690,"num_promote":11225,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1247440896,"data_stored":1247159944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40740568208,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1912971,"num_read_kb":7479700,"num_write":1888261,"num_write_kb":76299703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16636,"num_omap_keys":310,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44169244672,"data_stored":49438308640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80439,"ondisk_log_size":80439,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739305,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10264728,"kb_used_data":8649872,"kb_used_omap":376,"kb_used_meta":1614471,"kb_avail":84107112,"statfs":{"total":96636764160,"available":86125682688,"internally_reserved":0,"allocated":8857468928,"data_stored":9949775849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1653219229},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:30:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.76200000000000001,"15min":0.87},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.399,"5min":7.4219999999999997,"15min":7.907},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.0169999999999999,"15min":0.98299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":4.6180000000000003,"5min":9.6280000000000001,"15min":10.976000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Tue Oct 3 21:30:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.3859999999999999,"15min":1.1699999999999999},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":4.1230000000000002,"5min":9.9719999999999995,"15min":30.677},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.3220000000000001,"15min":1.3879999999999999},"min":{"1min":0.26000000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":6.2009999999999996,"5min":14.381,"15min":30.045999999999999},"last":0.52000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739294,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23952484,"kb_used_data":21870620,"kb_used_omap":963,"kb_used_meta":2080892,"kb_avail":70419356,"statfs":{"total":96636764160,"available":72109420544,"internally_reserved":0,"allocated":22395514880,"data_stored":25029599863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":986730,"internal_metadata":2130833814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,11,20,16,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:30:43 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.819,"15min":1.732},"min":{"1min":0.20100000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.7440000000000002,"5min":15.488,"15min":17.309999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.2529999999999999,"15min":1.1699999999999999},"min":{"1min":0.20499999999999999,"5min":0.19500000000000001,"15min":0.183},"max":{"1min":13.489000000000001,"5min":13.489000000000001,"15min":28.699999999999999},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Oct 3 21:30:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.577,"15min":1.419},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":5.0060000000000002,"5min":11.345000000000001,"15min":28.585999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.536,"15min":1.395},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":6.7089999999999996,"5min":9.5570000000000004,"15min":14.201000000000001},"last":0.45400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15725164,"kb_used_data":13826276,"kb_used_omap":597,"kb_used_meta":1898282,"kb_avail":78646676,"statfs":{"total":96636764160,"available":80534196224,"internally_reserved":0,"allocated":14158106624,"data_stored":15699325119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1943841697},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,4,10,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:30:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.88,"15min":1.014},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":4.3680000000000003,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.311},{"interface":"front","average":{"1min":1.034,"5min":1.1220000000000001,"15min":1.194},"min":{"1min":0.26400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":5.9119999999999999,"5min":13.823,"15min":23.422000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:30:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.77200000000000002,"15min":0.99399999999999999},"min":{"1min":0.249,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":5.9589999999999996,"5min":6.0819999999999999,"15min":8.9510000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.375,"5min":0.90900000000000003,"15min":0.96599999999999997},"min":{"1min":0.27700000000000002,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":5.9870000000000001,"5min":7.359,"15min":11.329000000000001},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13851684864,"data_stored":15393331112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22084485120,"data_stored":24719047824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8233074688,"data_stored":9325929704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305463296,"data_stored":305393466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309874688,"data_stored":309804042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":632102912,"data_stored":631962436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:31:35.541 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:31:35.542 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:31:38.497 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.489+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:31:38.502 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.493+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:31:38.506 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.501+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:31:38.507 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.501+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:31:38.507 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.501+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:31:38.514 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.509+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:31:38.514 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:38.509+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:31:38.609 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:38.605+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a680 2023-10-03T21:31:38.638 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:38.633+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:31:38.648 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:38.645+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c028f0 2023-10-03T21:31:38.674 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:38.669+0000 7f27ba5d7640 -1 reset not still connected to 0x556915c03040 2023-10-03T21:31:38.674 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:38.669+0000 7f27ba5d7640 -1 reset not still connected to 0x556915e7edd0 2023-10-03T21:31:38.674 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:31:38.669+0000 7f27ba5d7640 -1 reset not still connected to 0x5569166bf380 2023-10-03T21:31:39.945 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:39.941+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:31:39.968 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:39.957+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:31:48.424 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:48.417+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:31:48.425 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:48.417+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:31:48.430 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:31:48.425+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:31:55.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:31:56.292 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:31:56.293 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:31:56.365 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2456,"stamp":"2023-10-03T21:31:54.323942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41400965720,"num_objects":14926,"num_object_clones":9472,"num_object_copies":29852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":2005032,"num_read_kb":7745353,"num_write":1962248,"num_write_kb":76884283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63755,"num_flush":17197,"num_flush_kb":69305009,"num_evict":11234,"num_evict_kb":45352878,"num_promote":11276,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30482,"num_omap_keys":527,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120687,"ondisk_log_size":120687,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49592580,"kb_used_data":44405860,"kb_used_omap":2414,"kb_used_meta":5184273,"kb_avail":233522940,"statfs":{"total":289910292480,"available":239127490560,"internally_reserved":0,"allocated":45471600640,"data_stored":50724872176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2472680,"internal_metadata":5308695832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,7,8,1,4,31,7,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48394266,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":3978,"num_read_kb":15748,"num_write":3999,"num_write_kb":191409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":26,"num_flush":41,"num_flush_kb":167564,"num_evict":20,"num_evict_kb":81748,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":-1,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001662"},"pg_stats":[{"pgid":"2.7","version":"101'232786","reported_seq":465812,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.793391+0000","last_change":"2023-10-03T21:30:57.545790+0000","last_active":"2023-10-03T21:31:49.793391+0000","last_peered":"2023-10-03T21:31:49.793391+0000","last_clean":"2023-10-03T21:31:49.793391+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:31:49.793391+0000","last_undegraded":"2023-10-03T21:31:49.793391+0000","last_fullsized":"2023-10-03T21:31:49.793391+0000","mapping_epoch":12,"log_start":"97'222715","ondisk_log_start":"97'222715","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'230269","last_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_deep_scrub":"99'230269","last_deep_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_clean_scrub_stamp":"2023-10-03T21:30:57.545623+0000","objects_scrubbed":1786,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":188,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:27:25.972550+0000","scrub_duration":187.567402794,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5093396814,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":232491,"num_read_kb":909626,"num_write":230391,"num_write_kb":11081290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'233342","reported_seq":468868,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.727419+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:31:49.727419+0000","last_peered":"2023-10-03T21:31:49.727419+0000","last_clean":"2023-10-03T21:31:49.727419+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:31:49.727419+0000","last_undegraded":"2023-10-03T21:31:49.727419+0000","last_fullsized":"2023-10-03T21:31:49.727419+0000","mapping_epoch":12,"log_start":"97'223314","ondisk_log_start":"97'223314","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5142204735,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":235333,"num_read_kb":916972,"num_write":230885,"num_write_kb":10583980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'253313","reported_seq":507731,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T21:31:49.743079+0000","last_change":"2023-10-03T21:28:06.024812+0000","last_active":"2023-10-03T21:31:49.743079+0000","last_peered":"2023-10-03T21:31:49.743079+0000","last_clean":"2023-10-03T21:31:49.743079+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:31:49.743079+0000","last_undegraded":"2023-10-03T21:31:49.743079+0000","last_fullsized":"2023-10-03T21:31:49.743079+0000","mapping_epoch":12,"log_start":"97'243312","ondisk_log_start":"97'243312","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T20:13:26.103968+0000","last_clean_scrub_stamp":"2023-10-03T20:13:26.103968+0000","objects_scrubbed":1863,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 223s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5351141879,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":253888,"num_read_kb":994441,"num_write":250775,"num_write_kb":10022453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"101'241581","reported_seq":485254,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.727934+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:31:49.727934+0000","last_peered":"2023-10-03T21:31:49.727934+0000","last_clean":"2023-10-03T21:31:49.727934+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:31:49.727934+0000","last_undegraded":"2023-10-03T21:31:49.727934+0000","last_fullsized":"2023-10-03T21:31:49.727934+0000","mapping_epoch":12,"log_start":"97'231514","ondisk_log_start":"97'231514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5120623075,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":243067,"num_read_kb":949402,"num_write":239051,"num_write_kb":9146673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"101'37111","reported_seq":60992,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.381494+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:31:49.381494+0000","last_peered":"2023-10-03T21:31:49.381494+0000","last_clean":"2023-10-03T21:31:49.381494+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:31:49.381494+0000","last_undegraded":"2023-10-03T21:31:49.381494+0000","last_fullsized":"2023-10-03T21:31:49.381494+0000","mapping_epoch":15,"log_start":"84'27029","ondisk_log_start":"84'27029","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158936522,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":19882,"num_read_kb":59211,"num_write":17183,"num_write_kb":57132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15752,"num_flush":5025,"num_flush_kb":20294276,"num_evict":2753,"num_evict_kb":11135113,"num_promote":2761,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"101'232625","reported_seq":466052,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.589892+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:31:49.589892+0000","last_peered":"2023-10-03T21:31:49.589892+0000","last_clean":"2023-10-03T21:31:49.589892+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:31:49.589892+0000","last_undegraded":"2023-10-03T21:31:49.589892+0000","last_fullsized":"2023-10-03T21:31:49.589892+0000","mapping_epoch":12,"log_start":"97'222611","ondisk_log_start":"97'222611","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4834636097,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":232875,"num_read_kb":910013,"num_write":230256,"num_write_kb":7989388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"101'37017","reported_seq":60199,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.352180+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:31:49.352180+0000","last_peered":"2023-10-03T21:31:49.352180+0000","last_clean":"2023-10-03T21:31:49.352180+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:31:49.352180+0000","last_undegraded":"2023-10-03T21:31:49.352180+0000","last_fullsized":"2023-10-03T21:31:49.352180+0000","mapping_epoch":15,"log_start":"84'26935","ondisk_log_start":"84'26935","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":154943159,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":20723,"num_read_kb":56766,"num_write":16076,"num_write_kb":52163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15854,"num_flush":3802,"num_flush_kb":15294084,"num_evict":2869,"num_evict_kb":11554432,"num_promote":2882,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"101'244589","reported_seq":490663,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:50.713969+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:31:50.713969+0000","last_peered":"2023-10-03T21:31:50.713969+0000","last_clean":"2023-10-03T21:31:50.713969+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:31:50.713969+0000","last_undegraded":"2023-10-03T21:31:50.713969+0000","last_fullsized":"2023-10-03T21:31:50.713969+0000","mapping_epoch":12,"log_start":"97'234515","ondisk_log_start":"97'234515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5230809509,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":245511,"num_read_kb":961006,"num_write":242023,"num_write_kb":8819155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'32604","reported_seq":54096,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.347959+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:31:49.347959+0000","last_peered":"2023-10-03T21:31:49.347959+0000","last_clean":"2023-10-03T21:31:49.347959+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:31:49.347959+0000","last_undegraded":"2023-10-03T21:31:49.347959+0000","last_fullsized":"2023-10-03T21:31:49.347959+0000","mapping_epoch":15,"log_start":"84'22540","ondisk_log_start":"84'22540","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154796453,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":17817,"num_read_kb":56724,"num_write":16411,"num_write_kb":53963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16554,"num_flush":4191,"num_flush_kb":16925100,"num_evict":2828,"num_evict_kb":11456861,"num_promote":2839,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"101'235330","reported_seq":472942,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:50.669095+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:31:50.669095+0000","last_peered":"2023-10-03T21:31:50.669095+0000","last_clean":"2023-10-03T21:31:50.669095+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:31:50.669095+0000","last_undegraded":"2023-10-03T21:31:50.669095+0000","last_fullsized":"2023-10-03T21:31:50.669095+0000","mapping_epoch":12,"log_start":"97'225315","ondisk_log_start":"97'225315","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4944933315,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":236930,"num_read_kb":926858,"num_write":232929,"num_write_kb":7986301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'45976","reported_seq":74238,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.624462+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:31:49.624462+0000","last_peered":"2023-10-03T21:31:49.624462+0000","last_clean":"2023-10-03T21:31:49.624462+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:31:49.624462+0000","last_undegraded":"2023-10-03T21:31:49.624462+0000","last_fullsized":"2023-10-03T21:31:49.624462+0000","mapping_epoch":15,"log_start":"84'35932","ondisk_log_start":"84'35932","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":159079810,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":25856,"num_read_kb":62461,"num_write":16426,"num_write_kb":53692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15595,"num_flush":4179,"num_flush_kb":16791549,"num_evict":2784,"num_evict_kb":11206472,"num_promote":2794,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'242155","reported_seq":483300,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:31:49.770493+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:31:49.770493+0000","last_peered":"2023-10-03T21:31:49.770493+0000","last_clean":"2023-10-03T21:31:49.770493+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:31:49.770493+0000","last_undegraded":"2023-10-03T21:31:49.770493+0000","last_fullsized":"2023-10-03T21:31:49.770493+0000","mapping_epoch":12,"log_start":"97'232109","ondisk_log_start":"97'232109","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5054873984,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":240553,"num_read_kb":941782,"num_write":239705,"num_write_kb":11036199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":276,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-03T21:31:31.961705+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:31:31.961705+0000","last_peered":"2023-10-03T21:31:31.961705+0000","last_clean":"2023-10-03T21:31:31.961705+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:31:31.961705+0000","last_undegraded":"2023-10-03T21:31:31.961705+0000","last_fullsized":"2023-10-03T21:31:31.961705+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":627755944,"num_objects":231,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":84278,"num_read_kb":235162,"num_write":66096,"num_write_kb":216950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63755,"num_flush":17197,"num_flush_kb":69305009,"num_evict":11234,"num_evict_kb":45352878,"num_promote":11276,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1243193344,"data_stored":1242912592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40272,"ondisk_log_size":40272,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40772619408,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1920648,"num_read_kb":7510100,"num_write":1896015,"num_write_kb":76665439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16636,"num_omap_keys":310,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44233781248,"data_stored":49488607520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80316,"ondisk_log_size":80316,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10280572,"kb_used_data":8651828,"kb_used_omap":376,"kb_used_meta":1628359,"kb_avail":84091268,"statfs":{"total":96636764160,"available":86109458432,"internally_reserved":0,"allocated":8859471872,"data_stored":9950192720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1667440541},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:31:33 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.81100000000000005,"15min":0.90400000000000003},"min":{"1min":0.192,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":6.0519999999999996,"5min":7.4219999999999997,"15min":7.907},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.998,"5min":1.2310000000000001,"15min":1.022},"min":{"1min":0.20899999999999999,"5min":0.20799999999999999,"15min":0.183},"max":{"1min":18.623000000000001,"5min":18.623000000000001,"15min":18.623000000000001},"last":1.548}]},{"osd":1,"last update":"Tue Oct 3 21:31:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":1.2749999999999999,"15min":1.1699999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20000000000000001},"max":{"1min":2.0110000000000001,"5min":9.9719999999999995,"15min":30.677},"last":0.253},{"interface":"front","average":{"1min":2.5680000000000001,"5min":1.6839999999999999,"15min":1.5089999999999999},"min":{"1min":0.247,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":11.279999999999999,"5min":14.381,"15min":30.045999999999999},"last":5.585}]}]},{"osd":1,"up_from":8,"seq":34359739298,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23547376,"kb_used_data":21906336,"kb_used_omap":1441,"kb_used_meta":1639582,"kb_avail":70824464,"statfs":{"total":96636764160,"available":72524251136,"internally_reserved":0,"allocated":22432088064,"data_stored":25059024553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1476134,"internal_metadata":1678932442},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,0,3,28,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:31:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.5960000000000001,"15min":1.694},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.16200000000000001},"max":{"1min":5.7270000000000003,"5min":15.488,"15min":17.309999999999999},"last":0.251},{"interface":"front","average":{"1min":1.254,"5min":1.266,"15min":1.1899999999999999},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.189},"max":{"1min":5.7789999999999999,"5min":13.489000000000001,"15min":28.699999999999999},"last":0.38}]},{"osd":2,"last update":"Tue Oct 3 21:31:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.6970000000000001,"15min":1.4259999999999999},"min":{"1min":0.26000000000000001,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":5.8029999999999999,"5min":11.345000000000001,"15min":28.585999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.343,"5min":1.655,"15min":1.3999999999999999},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":6.7469999999999999,"5min":9.5570000000000004,"15min":14.201000000000001},"last":4.1989999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15764632,"kb_used_data":13847696,"kb_used_omap":597,"kb_used_meta":1916330,"kb_avail":78607208,"statfs":{"total":96636764160,"available":80493780992,"internally_reserved":0,"allocated":14180040704,"data_stored":15715654903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1962322849},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,0,1,3,3,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:30:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.88,"15min":1.014},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":4.3680000000000003,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.034,"5min":1.1220000000000001,"15min":1.194},"min":{"1min":0.26400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":5.9119999999999999,"5min":13.823,"15min":23.422000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:30:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.77200000000000002,"15min":0.99399999999999999},"min":{"1min":0.249,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":5.9589999999999996,"5min":6.0819999999999999,"15min":8.9510000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.375,"5min":0.90900000000000003,"15min":0.96599999999999997},"min":{"1min":0.27700000000000002,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":5.9870000000000001,"5min":7.359,"15min":11.329000000000001},"last":0.41199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13873561600,"data_stored":15409665960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22116798464,"data_stored":24744217744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8243421184,"data_stored":9334723816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313798656,"data_stored":313728879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314093568,"data_stored":314022969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":615301120,"data_stored":615160744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:31:56.367 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T20:13:26.103968+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=20, tm_min=13, tm_sec=26, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=21, tm_min=20, tm_sec=5, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T21:31:56.368 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T21:32:01.121 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:01.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:32:01.121 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:01.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:32:01.121 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:01.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259f1e0 2023-10-03T21:32:01.121 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:01.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:32:01.122 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:01.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:32:01.122 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:01.113+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:32:02.352 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-03T21:32:02.345+0000 7f27ba5d7640 -1 reset not still connected to 0x55691584a750 2023-10-03T21:32:06.294 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:06.289+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:32:06.308 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:06.301+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:32:06.309 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:06.301+0000 7fb3de4dc640 -1 reset not still connected to 0x55c69259edd0 2023-10-03T21:32:06.312 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:06.309+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cc000 2023-10-03T21:32:06.322 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:06.317+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cd6c0 2023-10-03T21:32:06.322 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:06.317+0000 7fb3de4dc640 -1 reset not still connected to 0x55c6932cdad0 2023-10-03T21:32:13.099 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:13.093+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc0820 2023-10-03T21:32:13.109 INFO:tasks.ceph.osd.0.smithi167.stderr:2023-10-03T21:32:13.105+0000 7fb3de4dc640 -1 reset not still connected to 0x55c691dc1040 2023-10-03T21:32:16.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T21:32:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-03T21:32:17.087 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2023-10-03T21:32:17.143 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":2467,"stamp":"2023-10-03T21:32:16.333085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41430555206,"num_objects":14924,"num_object_clones":9472,"num_object_copies":29848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14791,"num_whiteouts":11,"num_read":2015321,"num_read_kb":7786161,"num_write":1972547,"num_write_kb":77345624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63737,"num_flush":17302,"num_flush_kb":69734013,"num_evict":11298,"num_evict_kb":45614482,"num_promote":11338,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30989,"num_omap_keys":537,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120730,"ondisk_log_size":120730,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":49835684,"kb_used_data":44482380,"kb_used_omap":2414,"kb_used_meta":5350865,"kb_avail":233279836,"statfs":{"total":289910292480,"available":238878552064,"internally_reserved":0,"allocated":45549957120,"data_stored":50788057036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2472680,"internal_metadata":5479286040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,12,6,21,9,26,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44322773,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":6148,"num_read_kb":24368,"num_write":6128,"num_write_kb":281531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":-43,"num_flush":65,"num_flush_kb":265592,"num_evict":34,"num_evict_kb":138896,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002103"},"pg_stats":[{"pgid":"2.7","version":"101'233942","reported_seq":468142,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.319068+0000","last_change":"2023-10-03T21:30:57.545790+0000","last_active":"2023-10-03T21:32:15.319068+0000","last_peered":"2023-10-03T21:32:15.319068+0000","last_clean":"2023-10-03T21:32:15.319068+0000","last_became_active":"2023-10-03T20:13:27.128915+0000","last_became_peered":"2023-10-03T20:13:27.128915+0000","last_unstale":"2023-10-03T21:32:15.319068+0000","last_undegraded":"2023-10-03T21:32:15.319068+0000","last_fullsized":"2023-10-03T21:32:15.319068+0000","mapping_epoch":12,"log_start":"97'223915","ondisk_log_start":"97'223915","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'230269","last_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_deep_scrub":"99'230269","last_deep_scrub_stamp":"2023-10-03T21:30:57.545623+0000","last_clean_scrub_stamp":"2023-10-03T21:30:57.545623+0000","objects_scrubbed":1786,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":188,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:27:25.972550+0000","scrub_duration":187.567402794,"objects_trimmed":0,"snaptrim_duration":0.100532251,"stat_sum":{"num_bytes":5100622158,"num_objects":1786,"num_object_clones":1153,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":233657,"num_read_kb":914246,"num_write":231547,"num_write_kb":11163486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"101'234543","reported_seq":471294,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.337221+0000","last_change":"2023-10-03T21:28:15.214272+0000","last_active":"2023-10-03T21:32:15.337221+0000","last_peered":"2023-10-03T21:32:15.337221+0000","last_clean":"2023-10-03T21:32:15.337221+0000","last_became_active":"2023-10-03T20:13:27.128525+0000","last_became_peered":"2023-10-03T20:13:27.128525+0000","last_unstale":"2023-10-03T21:32:15.337221+0000","last_undegraded":"2023-10-03T21:32:15.337221+0000","last_fullsized":"2023-10-03T21:32:15.337221+0000","mapping_epoch":12,"log_start":"97'224514","ondisk_log_start":"97'224514","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'223453","last_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_deep_scrub":"97'223453","last_deep_scrub_stamp":"2023-10-03T21:28:15.214071+0000","last_clean_scrub_stamp":"2023-10-03T21:28:15.214071+0000","objects_scrubbed":1808,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":179,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:07:11.094117+0000","scrub_duration":178.32347521299999,"objects_trimmed":0,"snaptrim_duration":0.032834319000000001,"stat_sum":{"num_bytes":5143859519,"num_objects":1808,"num_object_clones":1171,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":236556,"num_read_kb":921820,"num_write":232086,"num_write_kb":10666324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2134,"num_omap_keys":54,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'254688","reported_seq":510414,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.336153+0000","last_change":"2023-10-03T21:31:56.681655+0000","last_active":"2023-10-03T21:32:15.336153+0000","last_peered":"2023-10-03T21:32:15.336153+0000","last_clean":"2023-10-03T21:32:15.336153+0000","last_became_active":"2023-10-03T20:13:27.128545+0000","last_became_peered":"2023-10-03T20:13:27.128545+0000","last_unstale":"2023-10-03T21:32:15.336153+0000","last_undegraded":"2023-10-03T21:32:15.336153+0000","last_fullsized":"2023-10-03T21:32:15.336153+0000","mapping_epoch":12,"log_start":"98'244612","ondisk_log_start":"98'244612","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'253706","last_scrub_stamp":"2023-10-03T21:31:56.681430+0000","last_deep_scrub":"101'253706","last_deep_scrub_stamp":"2023-10-03T21:31:56.681430+0000","last_clean_scrub_stamp":"2023-10-03T21:31:56.681430+0000","objects_scrubbed":1934,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":231,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:10:37.548755+0000","scrub_duration":230.56015296300001,"objects_trimmed":0,"snaptrim_duration":0.096349253999999995,"stat_sum":{"num_bytes":5356179959,"num_objects":1934,"num_object_clones":1244,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":255175,"num_read_kb":999561,"num_write":252099,"num_write_kb":10113325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":507,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"101'242803","reported_seq":487773,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.283118+0000","last_change":"2023-10-03T21:23:57.575281+0000","last_active":"2023-10-03T21:32:15.283118+0000","last_peered":"2023-10-03T21:32:15.283118+0000","last_clean":"2023-10-03T21:32:15.283118+0000","last_became_active":"2023-10-03T20:13:27.131521+0000","last_became_peered":"2023-10-03T20:13:27.131521+0000","last_unstale":"2023-10-03T21:32:15.283118+0000","last_undegraded":"2023-10-03T21:32:15.283118+0000","last_fullsized":"2023-10-03T21:32:15.283118+0000","mapping_epoch":12,"log_start":"97'232714","ondisk_log_start":"97'232714","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'219878","last_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_deep_scrub":"93'219878","last_deep_scrub_stamp":"2023-10-03T21:23:57.575082+0000","last_clean_scrub_stamp":"2023-10-03T21:23:57.575082+0000","objects_scrubbed":1879,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":215,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:37:21.316758+0000","scrub_duration":214.374899473,"objects_trimmed":0,"snaptrim_duration":0.0074964879999999999,"stat_sum":{"num_bytes":5128176099,"num_objects":1879,"num_object_clones":1213,"num_object_copies":3758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":244351,"num_read_kb":954482,"num_write":240273,"num_write_kb":9208769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2703,"num_omap_keys":73,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"101'37295","reported_seq":61289,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:14.524679+0000","last_change":"2023-10-03T21:20:19.999877+0000","last_active":"2023-10-03T21:32:14.524679+0000","last_peered":"2023-10-03T21:32:14.524679+0000","last_clean":"2023-10-03T21:32:14.524679+0000","last_became_active":"2023-10-03T20:13:32.176128+0000","last_became_peered":"2023-10-03T20:13:32.176128+0000","last_unstale":"2023-10-03T21:32:14.524679+0000","last_undegraded":"2023-10-03T21:32:14.524679+0000","last_fullsized":"2023-10-03T21:32:14.524679+0000","mapping_epoch":15,"log_start":"84'27229","ondisk_log_start":"84'27229","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'32167","last_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_deep_scrub":"90'32167","last_deep_scrub_stamp":"2023-10-03T21:20:19.999580+0000","last_clean_scrub_stamp":"2023-10-03T21:20:19.999580+0000","objects_scrubbed":56,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:35:21.189550+0000","scrub_duration":8.9137078209999991,"objects_trimmed":0,"snaptrim_duration":0.0037607140000000001,"stat_sum":{"num_bytes":158985683,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":19954,"num_read_kb":59499,"num_write":17274,"num_write_kb":57442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15761,"num_flush":5051,"num_flush_kb":20400468,"num_evict":2766,"num_evict_kb":11188217,"num_promote":2774,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2900,"num_omap_keys":44,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"101'233829","reported_seq":468474,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:14.843865+0000","last_change":"2023-10-03T21:24:20.306428+0000","last_active":"2023-10-03T21:32:14.843865+0000","last_peered":"2023-10-03T21:32:14.843865+0000","last_clean":"2023-10-03T21:32:14.843865+0000","last_became_active":"2023-10-03T20:13:27.127430+0000","last_became_peered":"2023-10-03T20:13:27.127430+0000","last_unstale":"2023-10-03T21:32:14.843865+0000","last_undegraded":"2023-10-03T21:32:14.843865+0000","last_fullsized":"2023-10-03T21:32:14.843865+0000","mapping_epoch":12,"log_start":"97'223811","ondisk_log_start":"97'223811","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'212765","last_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_deep_scrub":"93'212765","last_deep_scrub_stamp":"2023-10-03T21:24:20.305186+0000","last_clean_scrub_stamp":"2023-10-03T21:24:20.305186+0000","objects_scrubbed":1770,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":252,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:30.101599+0000","scrub_duration":251.14645609600001,"objects_trimmed":0,"snaptrim_duration":0.002572473,"stat_sum":{"num_bytes":4842115393,"num_objects":1770,"num_object_clones":1135,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":234088,"num_read_kb":914821,"num_write":231460,"num_write_kb":8047324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6001,"num_omap_keys":95,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"101'37204","reported_seq":60497,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:14.970590+0000","last_change":"2023-10-03T21:25:10.151617+0000","last_active":"2023-10-03T21:32:14.970590+0000","last_peered":"2023-10-03T21:32:14.970590+0000","last_clean":"2023-10-03T21:32:14.970590+0000","last_became_active":"2023-10-03T20:13:32.177077+0000","last_became_peered":"2023-10-03T20:13:32.177077+0000","last_unstale":"2023-10-03T21:32:14.970590+0000","last_undegraded":"2023-10-03T21:32:14.970590+0000","last_fullsized":"2023-10-03T21:32:14.970590+0000","mapping_epoch":15,"log_start":"84'27135","ondisk_log_start":"84'27135","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34134","last_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_deep_scrub":"94'34134","last_deep_scrub_stamp":"2023-10-03T21:25:10.150113+0000","last_clean_scrub_stamp":"2023-10-03T21:25:10.150113+0000","objects_scrubbed":60,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:11:50.421820+0000","scrub_duration":31.510674323,"objects_trimmed":0,"snaptrim_duration":0.098425096000000004,"stat_sum":{"num_bytes":150752950,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":20788,"num_read_kb":57026,"num_write":16169,"num_write_kb":52445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15853,"num_flush":3824,"num_flush_kb":15384080,"num_evict":2891,"num_evict_kb":11644440,"num_promote":2903,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3623,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"101'245824","reported_seq":493163,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.877425+0000","last_change":"2023-10-03T21:24:18.085262+0000","last_active":"2023-10-03T21:32:15.877425+0000","last_peered":"2023-10-03T21:32:15.877425+0000","last_clean":"2023-10-03T21:32:15.877425+0000","last_became_active":"2023-10-03T20:13:27.129334+0000","last_became_peered":"2023-10-03T20:13:27.129334+0000","last_unstale":"2023-10-03T21:32:15.877425+0000","last_undegraded":"2023-10-03T21:32:15.877425+0000","last_fullsized":"2023-10-03T21:32:15.877425+0000","mapping_epoch":12,"log_start":"97'235815","ondisk_log_start":"97'235815","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'223321","last_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_deep_scrub":"93'223321","last_deep_scrub_stamp":"2023-10-03T21:24:18.084977+0000","last_clean_scrub_stamp":"2023-10-03T21:24:18.084977+0000","objects_scrubbed":1881,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":246,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:40:31.632498+0000","scrub_duration":245.31909222100001,"objects_trimmed":0,"snaptrim_duration":0.016611542,"stat_sum":{"num_bytes":5232529829,"num_objects":1881,"num_object_clones":1217,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":246765,"num_read_kb":965962,"num_write":243258,"num_write_kb":8832271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'32766","reported_seq":54355,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:14.702523+0000","last_change":"2023-10-03T21:20:44.580843+0000","last_active":"2023-10-03T21:32:14.702523+0000","last_peered":"2023-10-03T21:32:14.702523+0000","last_clean":"2023-10-03T21:32:14.702523+0000","last_became_active":"2023-10-03T20:13:32.176527+0000","last_became_peered":"2023-10-03T20:13:32.176527+0000","last_unstale":"2023-10-03T21:32:14.702523+0000","last_undegraded":"2023-10-03T21:32:14.702523+0000","last_fullsized":"2023-10-03T21:32:14.702523+0000","mapping_epoch":15,"log_start":"84'22740","ondisk_log_start":"84'22740","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'28434","last_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_deep_scrub":"90'28434","last_deep_scrub_stamp":"2023-10-03T21:20:44.580646+0000","last_clean_scrub_stamp":"2023-10-03T21:20:44.580646+0000","objects_scrubbed":58,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":32,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:14.274552+0000","scrub_duration":31.507648736,"objects_trimmed":0,"snaptrim_duration":0.0025107739999999999,"stat_sum":{"num_bytes":154804602,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":17878,"num_read_kb":56968,"num_write":16492,"num_write_kb":54218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":16511,"num_flush":4214,"num_flush_kb":17018944,"num_evict":2845,"num_evict_kb":11526301,"num_promote":2856,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1957,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"101'236615","reported_seq":475600,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.869732+0000","last_change":"2023-10-03T21:28:04.977594+0000","last_active":"2023-10-03T21:32:15.869732+0000","last_peered":"2023-10-03T21:32:15.869732+0000","last_clean":"2023-10-03T21:32:15.869732+0000","last_became_active":"2023-10-03T20:13:27.130217+0000","last_became_peered":"2023-10-03T20:13:27.130217+0000","last_unstale":"2023-10-03T21:32:15.869732+0000","last_undegraded":"2023-10-03T21:32:15.869732+0000","last_fullsized":"2023-10-03T21:32:15.869732+0000","mapping_epoch":12,"log_start":"97'226515","ondisk_log_start":"97'226515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'225096","last_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_deep_scrub":"97'225096","last_deep_scrub_stamp":"2023-10-03T21:28:04.976546+0000","last_clean_scrub_stamp":"2023-10-03T21:28:04.976546+0000","objects_scrubbed":1797,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":248,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:59:49.937307+0000","scrub_duration":247.13011442800001,"objects_trimmed":0,"snaptrim_duration":0.063263301999999993,"stat_sum":{"num_bytes":4950032835,"num_objects":1797,"num_object_clones":1156,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":238295,"num_read_kb":932274,"num_write":234214,"num_write_kb":8024141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1448,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"101'46197","reported_seq":74604,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.304761+0000","last_change":"2023-10-03T21:24:35.749345+0000","last_active":"2023-10-03T21:32:15.304761+0000","last_peered":"2023-10-03T21:32:15.304761+0000","last_clean":"2023-10-03T21:32:15.304761+0000","last_became_active":"2023-10-03T20:13:32.177149+0000","last_became_peered":"2023-10-03T20:13:32.177149+0000","last_unstale":"2023-10-03T21:32:15.304761+0000","last_undegraded":"2023-10-03T21:32:15.304761+0000","last_fullsized":"2023-10-03T21:32:15.304761+0000","mapping_epoch":15,"log_start":"84'36132","ondisk_log_start":"84'36132","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'42050","last_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_deep_scrub":"93'42050","last_deep_scrub_stamp":"2023-10-03T21:24:35.749176+0000","last_clean_scrub_stamp":"2023-10-03T21:24:35.749176+0000","objects_scrubbed":60,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:27:50.664638+0000","scrub_duration":6.9945611760000004,"objects_trimmed":0,"snaptrim_duration":0.01907416,"stat_sum":{"num_bytes":154922387,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":25928,"num_read_kb":62749,"num_write":16522,"num_write_kb":54026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":15612,"num_flush":4213,"num_flush_kb":16930521,"num_evict":2796,"num_evict_kb":11255524,"num_promote":2805,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5366,"num_omap_keys":86,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"101'243466","reported_seq":485844,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-03T21:32:15.302808+0000","last_change":"2023-10-03T21:27:49.583023+0000","last_active":"2023-10-03T21:32:15.302808+0000","last_peered":"2023-10-03T21:32:15.302808+0000","last_clean":"2023-10-03T21:32:15.302808+0000","last_became_active":"2023-10-03T20:13:27.131801+0000","last_became_peered":"2023-10-03T20:13:27.131801+0000","last_unstale":"2023-10-03T21:32:15.302808+0000","last_undegraded":"2023-10-03T21:32:15.302808+0000","last_fullsized":"2023-10-03T21:32:15.302808+0000","mapping_epoch":12,"log_start":"97'233409","ondisk_log_start":"97'233409","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'230978","last_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_deep_scrub":"97'230978","last_deep_scrub_stamp":"2023-10-03T21:27:49.582839+0000","last_clean_scrub_stamp":"2023-10-03T21:27:49.582839+0000","objects_scrubbed":1838,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":210,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:28:28.911126+0000","scrub_duration":209.96119177099999,"objects_trimmed":0,"snaptrim_duration":0.35257012100000001,"stat_sum":{"num_bytes":5056983424,"num_objects":1838,"num_object_clones":1183,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":241780,"num_read_kb":946662,"num_write":241016,"num_write_kb":11069959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1450,"num_omap_keys":22,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"94'99","reported_seq":276,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-03T21:31:31.961705+0000","last_change":"2023-10-03T21:25:11.903712+0000","last_active":"2023-10-03T21:31:31.961705+0000","last_peered":"2023-10-03T21:31:31.961705+0000","last_clean":"2023-10-03T21:31:31.961705+0000","last_became_active":"2023-10-03T20:13:22.537245+0000","last_became_peered":"2023-10-03T20:13:22.537245+0000","last_unstale":"2023-10-03T21:31:31.961705+0000","last_undegraded":"2023-10-03T21:31:31.961705+0000","last_fullsized":"2023-10-03T21:31:31.961705+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'99","last_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_deep_scrub":"94'99","last_deep_scrub_stamp":"2023-10-03T21:25:11.901428+0000","last_clean_scrub_stamp":"2023-10-03T21:25:11.901428+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:27.263317+0000","scrub_duration":0.071339017000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":619465622,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":11,"num_read":84548,"num_read_kb":236242,"num_write":66457,"num_write_kb":218131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":32,"num_bytes_hit_set_archive":63737,"num_flush":17302,"num_flush_kb":69734013,"num_evict":11298,"num_evict_kb":45614482,"num_promote":11338,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13846,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1239220224,"data_stored":1238939436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48642,"internal_metadata":0},"log_size":40226,"ondisk_log_size":40226,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":40810499216,"num_objects":14693,"num_object_clones":9472,"num_object_copies":29386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14693,"num_whiteouts":0,"num_read":1930667,"num_read_kb":7549828,"num_write":1905953,"num_write_kb":77125599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17143,"num_omap_keys":320,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44310597632,"data_stored":49549941024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41010,"internal_metadata":0},"log_size":80405,"ondisk_log_size":80405,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739314,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10329576,"kb_used_data":8664736,"kb_used_omap":376,"kb_used_meta":1664455,"kb_avail":84042264,"statfs":{"total":96636764160,"available":86059278336,"internally_reserved":0,"allocated":8872689664,"data_stored":9961882686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":385123,"internal_metadata":1704402845},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:31:33 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.81100000000000005,"15min":0.90400000000000003},"min":{"1min":0.192,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":6.0519999999999996,"5min":7.4219999999999997,"15min":7.907},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.998,"5min":1.2310000000000001,"15min":1.022},"min":{"1min":0.20899999999999999,"5min":0.20799999999999999,"15min":0.183},"max":{"1min":18.623000000000001,"5min":18.623000000000001,"15min":18.623000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Tue Oct 3 21:31:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":1.2749999999999999,"15min":1.1699999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.20000000000000001},"max":{"1min":2.0110000000000001,"5min":9.9719999999999995,"15min":30.677},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.5680000000000001,"5min":1.6839999999999999,"15min":1.5089999999999999},"min":{"1min":0.247,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":11.279999999999999,"5min":14.381,"15min":30.045999999999999},"last":0.69199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359739303,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23697820,"kb_used_data":21935956,"kb_used_omap":1441,"kb_used_meta":1760414,"kb_avail":70674020,"statfs":{"total":96636764160,"available":72370196480,"internally_reserved":0,"allocated":22462418944,"data_stored":25081589433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1476134,"internal_metadata":1802664410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,0,13,3,16,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 21:31:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.5960000000000001,"15min":1.694},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.16200000000000001},"max":{"1min":5.7270000000000003,"5min":15.488,"15min":17.309999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.266,"15min":1.1899999999999999},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.189},"max":{"1min":5.7789999999999999,"5min":13.489000000000001,"15min":28.699999999999999},"last":0.249}]},{"osd":2,"last update":"Tue Oct 3 21:31:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.6970000000000001,"15min":1.4259999999999999},"min":{"1min":0.26000000000000001,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":5.8029999999999999,"5min":11.345000000000001,"15min":28.585999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.343,"5min":1.655,"15min":1.3999999999999999},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":6.7469999999999999,"5min":9.5570000000000004,"15min":14.201000000000001},"last":2.9289999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739317,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":15808288,"kb_used_data":13881688,"kb_used_omap":597,"kb_used_meta":1925994,"kb_avail":78563552,"statfs":{"total":96636764160,"available":80449077248,"internally_reserved":0,"allocated":14214848512,"data_stored":15744584917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":611423,"internal_metadata":1972218785},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,5,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 21:31:53 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.86099999999999999,"15min":0.98299999999999998},"min":{"1min":0.19,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":4.4130000000000003,"5min":7.7690000000000001,"15min":9.8450000000000006},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.0049999999999999,"15min":1.153},"min":{"1min":0.23999999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.9939999999999998,"5min":11.349,"15min":23.422000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Tue Oct 3 21:31:53 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.73999999999999999,"15min":0.98299999999999998},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.187},"max":{"1min":5.367,"5min":5.9589999999999996,"15min":8.9510000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.81399999999999995,"15min":0.95899999999999996},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":5.2220000000000004,"5min":6.633,"15min":11.329000000000001},"last":3.9710000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13902876672,"data_stored":15432800168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14806,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22155337728,"data_stored":24775007376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24291,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8252383232,"data_stored":9342133480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1913,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313860096,"data_stored":313790285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8837,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305745920,"data_stored":305675337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":619614208,"data_stored":619473814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22479,"internal_metadata":0}]}} 2023-10-03T21:32:17.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-03T21:32:18.107 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-03T21:32:18.107 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-03T21:32:18.107 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-03T21:32:18.108 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T21:32:20.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-03T21:32:24.209 INFO:tasks.ceph.osd.0:Stopped 2023-10-03T21:32:24.209 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-03T21:32:24.210 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T21:32:25.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-03T21:32:25.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-03T21:32:30.311 INFO:tasks.ceph.osd.1:Stopped 2023-10-03T21:32:30.312 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-03T21:32:30.312 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T21:32:30.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-03T21:32:30.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-03T21:32:36.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-10-03T21:32:36.414 INFO:tasks.ceph.osd.2:Stopped 2023-10-03T21:32:36.414 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-03T21:32:36.415 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-03T21:32:36.415 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T21:32:36.446 INFO:tasks.ceph.mgr.x:Stopped 2023-10-03T21:32:36.446 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-03T21:32:36.447 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-03T21:32:36.447 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T21:32:36.505 INFO:tasks.ceph.mon.a:Stopped 2023-10-03T21:32:36.505 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-03T21:32:36.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 2023-10-03T21:32:36.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T21:32:36.588 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-03T21:32:39.524 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T21:32:39.525 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-03T21:32:39.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T21:32:39.817 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-03T21:32:40.046 INFO:tasks.ceph:Archiving mon data... 2023-10-03T21:32:40.048 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828/data/mon.a.tgz 2023-10-03T21:32:40.048 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2023-10-03T21:32:40.083 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.Y74e3uFnGP 2023-10-03T21:32:40.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Y74e3uFnGP 2023-10-03T21:32:40.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.Y74e3uFnGP 2023-10-03T21:32:40.332 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.Y74e3uFnGP is 499KB 2023-10-03T21:32:40.398 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.Y74e3uFnGP 2023-10-03T21:32:40.403 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-03T21:32:40.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-03T21:32:40.487 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-03T21:32:40.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-03T21:32:40.559 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-03T21:32:40.563 INFO:tasks.ceph:Archiving crash dumps... 2023-10-03T21:32:40.565 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828/remote/smithi167/crash 2023-10-03T21:32:40.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-03T21:32:40.614 INFO:tasks.ceph:Compressing logs... 2023-10-03T21:32:40.615 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-03T21:32:40.688 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42813.log 2023-10-03T21:32:40.688 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41993.log 2023-10-03T21:32:40.690 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42813.log.gz 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53048.log 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54643.log 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41993.log: /var/log/ceph/ceph-client.admin.53048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47891.log 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41993.log.gz 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53048.log.gz 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48348.log 2023-10-03T21:32:40.691 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54643.log.gz 2023-10-03T21:32:40.692 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43268.log 2023-10-03T21:32:40.692 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47891.log.gz 2023-10-03T21:32:40.695 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40471.log 2023-10-03T21:32:40.695 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48348.log.gz 2023-10-03T21:32:40.695 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46131.log 2023-10-03T21:32:40.695 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43268.log.gz 2023-10-03T21:32:40.695 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42671.log 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40471.log.gz 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49717.log 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46131.log.gz 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42671.log.gz 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49808.log 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44765.log 2023-10-03T21:32:40.696 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49717.log.gz 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53008.log 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49808.log.gz 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44765.log.gz 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53008.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53008.log.gz -- 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr: /var/log/ceph/ceph-client.admin.39165.log 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53066.log 2023-10-03T21:32:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2023-10-03T21:32:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39165.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2023-10-03T21:32:40.698 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2023-10-03T21:32:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53248.log 2023-10-03T21:32:40.706 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53066.log.gz 2023-10-03T21:32:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2023-10-03T21:32:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2023-10-03T21:32:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41344.log 2023-10-03T21:32:40.706 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53248.log.gz 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47606.log 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48317.log 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41344.log.gz 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51785.log 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47606.log.gz 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2023-10-03T21:32:40.707 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48317.log.gz 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46192.log 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51785.log.gz 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gz 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46966.log 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46192.log.gz 2023-10-03T21:32:40.708 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54695.log 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43983.log 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46966.log.gz 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53274.log 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54695.log.gz 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43983.log.gz 2023-10-03T21:32:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53274.log.gzgzip 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44240.log 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50080.log 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54785.log 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44240.log.gz 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42896.log 2023-10-03T21:32:40.710 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50080.log.gz 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54785.log.gz 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41302.log 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42896.log.gz 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39268.log 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2023-10-03T21:32:40.711 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41302.log.gz 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51622.log 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39268.log.gz 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44940.log 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51622.log.gz 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2023-10-03T21:32:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46349.log 2023-10-03T21:32:40.713 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2023-10-03T21:32:40.713 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2023-10-03T21:32:40.713 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2023-10-03T21:32:40.760 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50977.log 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46349.log.gz 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40724.log 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45036.log 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50977.log.gz 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53808.log 2023-10-03T21:32:40.761 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49094.log 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45036.log.gz 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46876.log 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53808.log.gz 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48687.log 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49094.log.gz 2023-10-03T21:32:40.762 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49561.log 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46876.log.gz 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48687.log.gz 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49561.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49561.log.gz 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47117.log 2023-10-03T21:32:40.763 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53626.log 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49124.log 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47117.log.gz 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43890.log 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53626.log.gz 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44455.log 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49124.log.gz 2023-10-03T21:32:40.764 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43890.log.gz 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48016.log 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44455.log.gz 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40093.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40940.log 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48016.log.gz 2023-10-03T21:32:40.765 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40940.log.gz 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54812.log 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44136.log 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51501.log 2023-10-03T21:32:40.766 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54812.log.gz 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44136.log.gz 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53730.log 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51501.log: -- /var/log/ceph/ceph-client.admin.54037.log 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51501.log.gz 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46690.log 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53730.log.gz 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2023-10-03T21:32:40.767 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54037.log.gz 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46690.log.gz 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53912.log 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2023-10-03T21:32:40.768 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41870.log 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52617.log 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53912.log.gz 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41870.log.gz 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54465.log 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47982.log 2023-10-03T21:32:40.769 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52617.log.gz 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43146.log 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54465.log.gz 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47982.log.gz 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50548.log 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46009.log 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43146.log.gz 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48196.log 2023-10-03T21:32:40.770 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50548.log.gz 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46009.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46009.log.gz -5 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41690.log 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48196.log.gz 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47420.log 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41690.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41690.log.gz 2023-10-03T21:32:40.771 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53196.log 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45683.log 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47420.log.gz 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40022.log 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53196.log.gz 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54436.log 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45683.log.gz 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52766.log 2023-10-03T21:32:40.772 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40022.log.gz 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38913.log 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54436.log.gz 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38801.log 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52766.log.gz 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52916.log 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38913.log.gz 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38801.log.gz 2023-10-03T21:32:40.773 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41092.log 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47270.log 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52916.log.gz 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45322.log 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41092.log.gz 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51007.log 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47270.log.gz 2023-10-03T21:32:40.774 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42993.log 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45322.log.gz 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40616.log 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51007.log.gz 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51103.log 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42993.log.gz 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45901.log 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2023-10-03T21:32:40.775 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48381.log 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51103.log.gz 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50822.log 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45901.log.gz 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44984.log 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48381.log.gz 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50822.log.gz 2023-10-03T21:32:40.776 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52712.log 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44984.log.gz 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52712.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30694.log 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52712.log.gz 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43177.log 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2023-10-03T21:32:40.777 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.tmp-client.admin.30694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44414.log 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30694.log.gz 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45374.log 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43177.log.gz 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43641.log 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44414.log.gz 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip/var/log/ceph/ceph-client.admin.45374.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45374.log.gz 2023-10-03T21:32:40.778 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34491.log 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43641.log.gz 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53599.log 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34491.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34491.log.gz 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40201.log 2023-10-03T21:32:40.779 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53599.log.gz 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48076.log 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52089.log 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40201.log.gz 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53782.log 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48076.log.gz 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51592.log 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52089.log.gz 2023-10-03T21:32:40.780 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43067.log 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53782.log.gz 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38689.log 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51592.log.gz 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43067.log.gz 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53084.log 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38689.log.gz 2023-10-03T21:32:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48502.log 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53084.log.gz 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47210.log 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48411.log 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48502.log.gz 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40309.log 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47210.log.gz 2023-10-03T21:32:40.782 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53966.log 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48411.log.gz 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39911.log 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40309.log.gz 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49748.log 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53966.log.gz 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52119.log 0.0% 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39911.log.gz 2023-10-03T21:32:40.783 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34429.log 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49748.log.gz 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43787.log 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52119.log.gz 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34429.log.gz 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51877.log 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53378.log 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43787.log.gz 2023-10-03T21:32:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47637.log 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51877.log.gz 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54758.log 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53378.log.gz 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47637.log.gz 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49032.log 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54758.log.gz 2023-10-03T21:32:40.785 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43669.log 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39068.log 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49032.log.gz 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50672.log 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43669.log.gz 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39068.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-03T21:32:40.786 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39068.log.gz 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40056.log 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50672.log.gz 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49408.log 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51073.log 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40056.log.gz 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51561.log/var/log/ceph/ceph-client.admin.49408.log: 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49408.log.gz 2023-10-03T21:32:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53756.log 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51073.log.gz 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51561.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48441.log 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51561.log.gz 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50854.log 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53756.log.gz 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48441.log.gz 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54491.log 2023-10-03T21:32:40.788 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50854.log.gz 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54491.log.gz 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54254.log 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54092.log 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54254.log.gz 2023-10-03T21:32:40.789 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54092.log.gz 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43862.log 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43862.log.gz 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46535.log 2023-10-03T21:32:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40130.log 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46535.log.gz 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42612.log 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53860.log 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40130.log.gz 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42612.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42612.log.gz -5 2023-10-03T21:32:40.791 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52880.log 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53860.log.gz 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52880.log.gz 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38991.log 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51817.log 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38991.log.gz 2023-10-03T21:32:40.792 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44321.log 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54148.log 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51817.log.gz 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44321.log.gz 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54838.log 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44793.log 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54148.log.gz 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54838.log.gz 2023-10-03T21:32:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52805.log 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42639.log 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44793.log.gz 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52805.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52805.log.gz 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50609.log 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42639.log.gz 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53030.log 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2023-10-03T21:32:40.794 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50609.log.gz 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53030.log.gz 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44737.log 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53652.log 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51223.log 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44737.log.gz 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2023-10-03T21:32:40.795 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53652.log.gz 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51223.log.gz 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44067.log 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47240.log 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44067.log.gz 2023-10-03T21:32:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42270.log 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47240.log.gz 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42865.log 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47360.log 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42270.log.gz 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40292.log 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42865.log.gz 2023-10-03T21:32:40.797 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47732.log 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47360.log.gz 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40508.log 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40292.log.gz 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46380.log 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47732.log.gz 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40508.log.gz 2023-10-03T21:32:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41806.log 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46380.log.gz 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52304.log 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49838.log 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41806.log.gz 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45553.log 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52304.log.gz 2023-10-03T21:32:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49838.log.gz 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47481.log 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53103.log 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45553.log.gz 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46475.log 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47481.log.gz 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53103.log.gz 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2023-10-03T21:32:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54306.log 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46475.log.gz 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41012.log 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54306.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.54306.log.gz /var/log/ceph/ceph-client.admin.48906.log 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51652.log 2023-10-03T21:32:40.801 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41012.log.gz 2023-10-03T21:32:40.900 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44106.log 2023-10-03T21:32:40.900 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48906.log.gz 2023-10-03T21:32:40.900 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40147.log 2023-10-03T21:32:40.900 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51652.log.gz 2023-10-03T21:32:40.900 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44106.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44106.log.gz 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr: /var/log/ceph/ceph-client.admin.46161.log 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40147.log.gz 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49976.log 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46161.log.gz 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz 2023-10-03T21:32:40.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47511.log 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48627.log 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49976.log.gz 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47511.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41949.log -- replaced with /var/log/ceph/ceph-client.admin.47511.log.gz 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50887.log 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48627.log.gz 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41217.log 2023-10-03T21:32:40.902 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41949.log.gz 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50792.log 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50887.log.gz 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41217.log.gz 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50792.log.gz 2023-10-03T21:32:40.903 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50397.log 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43818.log 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47762.log 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50397.log.gz 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48717.log 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43818.log.gz 2023-10-03T21:32:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47762.log.gz 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr:gzip/var/log/ceph/ceph-client.admin.48717.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45229.log 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48717.log.gz 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45257.log 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52647.log 2023-10-03T21:32:40.905 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45229.log.gz 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45257.log.gz 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52693.log 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52647.log.gz 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52934.log 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52693.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52693.log.gz /var/log/ceph/ceph-client.admin.54118.log 2023-10-03T21:32:40.906 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52934.log.gz 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54118.log.gz 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52461.log 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49186.log 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40434.log 2023-10-03T21:32:40.907 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52461.log.gz 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47300.log 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49186.log.gz 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53461.log 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40434.log.gz 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39965.log 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47300.log.gz 2023-10-03T21:32:40.908 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52784.log 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53461.log.gz 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39965.log.gz 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53300.log 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52784.log.gz 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2023-10-03T21:32:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51712.log 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53300.log.gz 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51288.log 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44661.log 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51712.log.gz 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45493.log 2023-10-03T21:32:40.910 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51288.log.gz 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52970.log 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44661.log.gz 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45493.log.gz 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52842.log 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52970.log.gz 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51193.log 2023-10-03T21:32:40.911 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45523.log 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52842.log.gz 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40110.log 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51193.log.gz 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45523.log.gz 2023-10-03T21:32:40.912 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52182.log 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40110.log.gz 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44264.log 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45651.log 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52182.log.gz 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5/var/log/ceph/ceph-client.admin.44264.log: --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44264.log.gz 2023-10-03T21:32:40.913 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49778.log 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45651.log.gz 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51255.log 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46039.log 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49778.log.gz 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46505.log 2023-10-03T21:32:40.914 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51255.log.gz 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48846.log 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46039.log.gz 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44540.log 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46505.log.gz 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49220.log 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48846.log.gz 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44540.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-03T21:32:40.915 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44540.log.gz 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log/var/log/ceph/ceph-client.admin.49220.log: 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49220.log.gz 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39985.log 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53678.log 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42560.log 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39985.log.gz 2023-10-03T21:32:40.916 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50579.log 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53678.log.gz 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48257.log 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42560.log.gz 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50579.log.gz 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40184.log 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49869.log 2023-10-03T21:32:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48257.log.gz 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46724.log 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40184.log.gz 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49869.log.gz 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48972.log 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46724.log.gz 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2023-10-03T21:32:40.918 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48972.log.gz 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50762.log 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50518.log 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2023-10-03T21:32:40.919 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44513.log 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50762.log.gz 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50518.log.gz 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44513.log.gz 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38853.log 2023-10-03T21:32:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47150.log 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38853.log.gz 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51908.log 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52860.log 2023-10-03T21:32:40.921 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47150.log.gz 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43572.log 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51908.log.gz 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52860.log.gz 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47057.log 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43572.log.gz 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48939.log 2023-10-03T21:32:40.922 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47057.log.gz 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46600.log 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48939.log.gz 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46565.log 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49901.log 2023-10-03T21:32:40.923 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46600.log.gz 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41428.log 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46565.log.gz 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52898.log 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49901.log.gz 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46101.log 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41428.log.gz 2023-10-03T21:32:40.924 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50118.log 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52898.log.gz 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46101.log.gz 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49592.log 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43237.log 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50118.log: /var/log/ceph/ceph-client.admin.49592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50118.log.gz 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49592.log.gz 2023-10-03T21:32:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43237.log.gz 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42769.log 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gz 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50245.log 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2023-10-03T21:32:40.926 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42769.log.gz 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49346.log 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50245.log.gz 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47390.log 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50367.log 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49346.log.gz 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49686.log 2023-10-03T21:32:40.927 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47390.log.gz 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44489.log 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50367.log.gz 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49686.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49686.log.gz 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44630.log 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43391.log 2023-10-03T21:32:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53222.log 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44630.log.gz 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52272.log 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43391.log.gz 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49501.log 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53222.log.gz 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34370.log 2023-10-03T21:32:40.929 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52272.log.gz 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43312.log 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34370.log.gz 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43542.log 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49250.log 2023-10-03T21:32:40.930 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43312.log.gz 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49626.log 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43542.log.gz 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49250.log.gz 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49626.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49626.log.gz 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2023-10-03T21:32:40.931 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47792.log 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52431.log 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47792.log.gz 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52730.log 2023-10-03T21:32:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52431.log.gz 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51938.log 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52730.log.gz 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47669.log 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51938.log.gz 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50153.log 2023-10-03T21:32:40.933 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47669.log.gz 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40164.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44007.log 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43340.log 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50153.log.gz 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53518.log 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44007.log.gz 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48226.log 2023-10-03T21:32:40.934 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43340.log.gz 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48046.log 0.0% 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53518.log.gz 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52212.log 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48226.log.gz 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48657.log 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48046.log.gz 2023-10-03T21:32:40.935 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52367.log 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52212.log.gz 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50307.log 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48657.log.gz 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52367.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52367.log.gz 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52587.log 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51531.log 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50307.log.gz 2023-10-03T21:32:40.936 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52587.log.gz 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39368.log 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51531.log.gz 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr: 89.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40633.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr: /var/log/ceph/ceph-client.admin.45743.log 2023-10-03T21:32:40.937 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53144.log 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39368.log.gz 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45743.log.gz 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53144.log.gz 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2023-10-03T21:32:40.938 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42703.log 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42136.log 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49376.log 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42703.log.gz 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45585.log 2023-10-03T21:32:40.939 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42136.log.gz 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51318.log 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49376.log.gz 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45585.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45585.log.gz -5 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42841.log 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54864.log 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51318.log.gz 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2023-10-03T21:32:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42841.log.gz 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45972.log 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54864.log.gz 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54358.log 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50458.log 2023-10-03T21:32:40.941 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47330.log 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54358.log.gz 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50458.log.gz 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51039.log 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47330.log.gz 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44167.log 2023-10-03T21:32:40.942 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42242.log 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51039.log.gz 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53490.log 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44167.log.gz 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48472.log 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42242.log.gz 2023-10-03T21:32:40.943 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53490.log.gz 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39928.log 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54410.log 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48472.log.gz 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49316.log 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39928.log.gz 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38879.log 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54410.log.gz 2023-10-03T21:32:40.944 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52823.log 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49316.log.gz 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49282.log 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38879.log.gz 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53170.log 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52823.log.gz 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49156.log 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49282.log.gz 2023-10-03T21:32:40.945 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53170.log.gz 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44850.log 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49156.log.gz 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50337.log 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47952.log 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44850.log.gz 2023-10-03T21:32:40.946 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53545.log 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50337.log.gz 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46936.log 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47952.log.gz 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53545.log.gz 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41922.log 2023-10-03T21:32:40.947 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46936.log.gz 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51682.log 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41922.log.gz 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51682.log.gz 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54554.log 2023-10-03T21:32:40.948 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43613.log 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54554.log.gz 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52557.log 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2023-10-03T21:32:40.949 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43613.log.gz 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53704.log 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52557.log.gz 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45093.log 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48136.log 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53704.log.gz 2023-10-03T21:32:40.950 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44600.log 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45093.log.gz 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54063.log 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48136.log.gz 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46785.log 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44600.log.gz 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40687.log 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54063.log.gz 2023-10-03T21:32:40.951 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38750.log 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46785.log.gz 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44880.log 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38750.log.gz 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2023-10-03T21:32:40.952 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48748.log 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44880.log.gz 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38983.log 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48748.log.gz 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47087.log 2023-10-03T21:32:40.953 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38983.log.gz 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40866.log 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40238.log 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47087.log.gz 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54228.log 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40866.log.gz 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40238.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-03T21:32:40.954 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40238.log.gz 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48532.log 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54228.log.gz 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47541.log 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48532.log.gz 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50642.log 2023-10-03T21:32:40.955 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52397.log 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47541.log.gz 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50642.log.gz 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53571.log 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52397.log.gz 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48786.log 2023-10-03T21:32:40.956 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53571.log.gz 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42733.log 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48786.log.gz 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45401.log 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2023-10-03T21:32:40.957 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51379.log 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42733.log.gz 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52028.log 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45401.log.gz 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46845.log 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51379.log.gz 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53404.log 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52028.log.gz 2023-10-03T21:32:40.958 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46754.log 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46845.log.gz 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53404.log.gz 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53121.log 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46996.log 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46754.log.gz 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49002.log 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53121.log.gz 2023-10-03T21:32:40.959 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54332.log 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46996.log.gz 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53352.log 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49002.log.gz 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54580.log 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54332.log.gz 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53352.log.gz 2023-10-03T21:32:40.960 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51348.log 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54580.log.gz 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39243.log 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51348.log.gz 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40076.log 2023-10-03T21:32:40.961 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39243.log.gz 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45869.log 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40076.log.gz 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45869.log: -- /var/log/ceph/ceph-client.admin.54888.log 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45869.log.gz 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50013.log 2023-10-03T21:32:40.962 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50702.log 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50013.log.gz 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48816.log 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50702.log.gz 2023-10-03T21:32:40.963 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51163.log 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48816.log.gz 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52988.log 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47026.log 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51163.log.gz 2023-10-03T21:32:40.964 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42168.log 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52988.log.gz 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54384.log 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47026.log.gz 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42168.log.gz 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51847.log 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54384.log.gz 2023-10-03T21:32:40.965 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gz 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52748.log 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51847.log.gz 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40994.log 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48106.log 2023-10-03T21:32:40.966 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52748.log.gz 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53326.log 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40994.log.gz 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51440.log 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48106.log.gz 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43955.log 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53326.log.gz 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53435.log 2023-10-03T21:32:40.967 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51440.log.gz 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43955.log.gz 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39069.log 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53435.log.gz 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52242.log 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45157.log 2023-10-03T21:32:40.968 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39069.log.gz 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52242.log.gz 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47822.log 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45157.log.gz 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53938.log 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2023-10-03T21:32:40.969 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50427.log 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47822.log.gz 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50183.log 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53938.log.gz 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48166.log 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50427.log.gz 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50183.log: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.45972.log.gz 2023-10-03T21:32:40.970 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48287.log 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50183.log.gz 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54607.log 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48166.log.gz 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48287.log.gz 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45350.log 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54607.log.gz 2023-10-03T21:32:40.971 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42072.log 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45350.log.gz 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34394.log 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42072.log.gz 2023-10-03T21:32:40.972 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46319.log 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34394.log.gz 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40670.log 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46319.log.gz 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2023-10-03T21:32:40.973 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40346.log 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40670.log.gz 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47451.log 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41470.log 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40346.log.gz 2023-10-03T21:32:40.974 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44910.log 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47451.log.gz 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41470.log: /var/log/ceph/ceph-client.admin.41260.log 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41470.log.gz 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45839.log 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44910.log.gz 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54517.log 2023-10-03T21:32:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54669.log 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45839.log.gz 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46288.log 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54517.log.gz 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49656.log 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54669.log.gz 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2023-10-03T21:32:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46288.log.gz 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48592.log 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49656.log.gz 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42351.log 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47180.log 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48592.log.gz 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54202.log 2023-10-03T21:32:40.977 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42351.log.gz 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40778.log 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47180.log.gz 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39945.log 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54202.log.gz 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51968.log 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40778.log.gz 2023-10-03T21:32:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.39945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48876.log 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39945.log.gz 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52058.log 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51968.log.gz 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.48876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38912.log 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48876.log.gz 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52058.log.gz 2023-10-03T21:32:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46660.log 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45713.log 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38912.log.gz 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41898.log 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46660.log.gz 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49439.log 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45713.log.gz 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41898.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41898.log.gz 2023-10-03T21:32:40.980 INFO:teuthology.orchestra.run.smithi167.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52675.log 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:gzip/var/log/ceph/ceph-client.admin.49439.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42321.log 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49439.log.gz 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43023.log 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52675.log.gz 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52952.log 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42321.log.gz 2023-10-03T21:32:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51410.log 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43023.log.gz 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54721.log 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52952.log.gz 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51998.log 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51410.log.gz 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52149.log 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54721.log.gz 2023-10-03T21:32:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49931.log 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51998.log.gz 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52149.log.gz 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49469.log 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49931.log.gz 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2023-10-03T21:32:40.983 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42926.log 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49469.log.gz 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46441.log 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42926.log.gz 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45187.log 2023-10-03T21:32:40.984 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46441.log.gz 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44570.log 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45187.log.gz 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45063.log 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.44570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50049.log 2023-10-03T21:32:40.985 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44570.log.gz 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45063.log.gz 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52491.log 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50049.log.gz 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51133.log 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2023-10-03T21:32:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42480.log 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52491.log.gz 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43207.log 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51133.log.gz 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42480.log.gz 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52334.log 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43207.log.gz 2023-10-03T21:32:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53886.log 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47701.log 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.52334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52334.log.gz 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47852.log 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53886.log.gz 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50275.log 2023-10-03T21:32:40.988 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47701.log.gz 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45012.log 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.47852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47852.log.gz 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43364.log 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50275.log.gz 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45012.log.gz 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54176.log 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45779.log 2023-10-03T21:32:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43364.log.gz 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.54176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51742.log 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54176.log.gz 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45779.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45779.log.gz -5 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53834.log 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.51742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50732.log 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51742.log.gz 2023-10-03T21:32:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.53834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46906.log 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53834.log.gz 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.50732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45462.log 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50732.log.gz 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.46906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46906.log.gz 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.45462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45462.log.gz 2023-10-03T21:32:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.41648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40579.log 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:gzip/var/log/ceph/ceph-client.admin.46255.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49531.log 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.40579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43757.log 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40579.log.gz 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.49531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49531.log.gz 2023-10-03T21:32:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.43757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43757.log.gz 2023-10-03T21:32:40.993 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.42208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2023-10-03T21:32:41.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-03T21:32:41.129 INFO:teuthology.orchestra.run.smithi167.stderr: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-03T21:32:50.191 INFO:teuthology.orchestra.run.smithi167.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-03T21:32:56.431 INFO:teuthology.orchestra.run.smithi167.stderr: 89.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-03T22:12:15.882 INFO:teuthology.orchestra.run.smithi167.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-03T22:22:09.130 INFO:teuthology.orchestra.run.smithi167.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-03T22:28:32.565 INFO:teuthology.orchestra.run.smithi167.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-03T22:28:32.566 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-03T22:28:32.566 INFO:teuthology.orchestra.run.smithi167.stderr:real 55m51.905s 2023-10-03T22:28:32.567 INFO:teuthology.orchestra.run.smithi167.stderr:user 46m56.618s 2023-10-03T22:28:32.567 INFO:teuthology.orchestra.run.smithi167.stderr:sys 2m20.737s 2023-10-03T22:28:32.568 INFO:tasks.ceph:Archiving logs... 2023-10-03T22:28:32.569 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828/remote/smithi167/log 2023-10-03T22:28:32.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-03T22:31:11.859 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-03T22:31:11.870 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-03T22:31:11.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-03T22:31:11.925 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-03T22:31:11.934 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-03T22:31:11.934 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T22:31:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2023-10-03T22:31:12.031 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2023-10-03T22:31:12.031 INFO:teuthology.orchestra.run.smithi167.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 256 377 0.182 +1.071 0.331 2023-10-03T22:31:12.031 INFO:teuthology.orchestra.run.smithi167.stdout:+hv02.front.sepi 63.231.80.2 3 u 64 256 377 0.197 +0.207 0.358 2023-10-03T22:31:12.031 INFO:teuthology.orchestra.run.smithi167.stdout:+hv03.front.sepi 74.6.168.72 3 u 22 256 377 0.184 -2.601 0.368 2023-10-03T22:31:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-03T22:31:12.032 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-03T22:31:12.042 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-03T22:31:12.042 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-03T22:31:12.051 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-03T22:31:12.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-03T22:31:12.068 INFO:teuthology.task.internal:Duration was 8919.432340 seconds 2023-10-03T22:31:12.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-03T22:31:12.077 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-03T22:31:12.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-03T22:31:12.343 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-03T22:31:12.343 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2023-10-03T22:31:12.343 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-03T22:31:12.377 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-03T22:31:12.378 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-03T22:31:12.490 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-03T22:31:12.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-03T22:31:12.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-03T22:31:12.860 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-03T22:31:12.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-03T22:31:12.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-03T22:31:12.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-03T22:31:12.953 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2023-10-03T22:31:12.979 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-03T22:31:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T22:31:13.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-03T22:31:13.043 INFO:teuthology.task.internal:Transferring archived files... 2023-10-03T22:31:13.045 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828/remote/smithi167 2023-10-03T22:31:13.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-03T22:31:13.097 INFO:teuthology.task.internal:Removing archive directory... 2023-10-03T22:31:13.097 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-03T22:31:13.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-03T22:31:13.155 INFO:teuthology.task.internal:Not uploading archives. 2023-10-03T22:31:13.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-03T22:31:13.219 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-03T22:31:13.219 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-03T22:31:13.251 INFO:teuthology.orchestra.run.smithi167.stdout: 655473 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 3 22:31 /home/ubuntu/cephtest 2023-10-03T22:31:13.252 INFO:teuthology.orchestra.run.smithi167.stdout: 673313 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 20:13 /home/ubuntu/cephtest/mnt.0 2023-10-03T22:31:13.252 INFO:teuthology.orchestra.run.smithi167.stdout: 673744 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 20:15 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-03T22:31:13.252 INFO:teuthology.orchestra.run.smithi167.stdout: 687626 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 3 20:15 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-03T22:31:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T22:31:13.253 INFO:teuthology.orchestra.run.smithi167.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-03T22:31:13.254 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T22:31:13.262 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-03T22:31:13.272 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-03T22:31:13.283 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi167 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T22:31:13.283 INFO:teuthology.nuke:Checking targets against current locks 2023-10-03T22:31:13.311 DEBUG:teuthology.nuke:shortname: smithi167 2023-10-03T22:31:13.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T22:31:13.332 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-03_19:52:41-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409828', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 19:53:35.524529', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5dENvZ8CjFh1oP9HEOOvUxtCMq4Isbx8hO++REGaHaB5KJwIPB64PWkzDAYeBogfIJrNMGYZDotx2PmdKyH+w='} 2023-10-03T22:31:13.383 INFO:teuthology.orchestra.console.smithi167:Power off 2023-10-03T22:31:13.384 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-03T22:31:13.408 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2023-10-03T22:31:13.409 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T22:31:13.432 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-03T22:31:17.433 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T22:31:17.543 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-03T22:31:21.546 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T22:31:21.656 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2023-10-03T22:31:21.756 INFO:teuthology.orchestra.console.smithi167:Power off completed 2023-10-03T22:31:21.862 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 8919.432339668274 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi167 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=076c84df32724e61ae8155dfd15f1868 status: fail success: false 2023-10-03T22:31:21.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T22:31:21.952 INFO:teuthology.run:FAIL